[14]11

# Statistical Inference for Diagnostic Test Accuracy Studies with Multiple
Comparisons

Max Westphal
Fraunhofer MEVIS
Institute for Digital Medicine
Bremen, Germany
&Antonia Zapf11footnotemark: 1
Department of Medical Biometry and Epidemiology
University Medical Center Hamburg-Eppendorf
Hamburg, Germany The two authors contributed equally and are listed in
alphabetical order.Corresponding author: Max Westphal,
<EMAIL_ADDRESS>https://orcid.org/0000-0002-8488-758X

###### Abstract

Diagnostic accuracy studies assess sensitivity and specificity of a new index
test in relation to an established comparator or the reference standard. The
development and selection of the index test is usually assumed to be conducted
prior to the accuracy study. In practice, this is often violated, for instance
if the choice of the (apparently) best biomarker, model or cutpoint is based
on the same data that is used later for validation purposes. In this work, we
investigate several multiple comparison procedures which provide family-wise
error rate control for the emerging multiple testing problem. Due to the
nature of the co-primary hypothesis problem, conventional approaches for
multiplicity adjustment are too conservative for the specific problem and thus
need to be adapted. In an extensive simulation study, five multiple comparison
procedures are compared with regards to statistical error rates in least-
favorable and realistic scenarios. This covers parametric and nonparamtric
methods and one Bayesian approach. All methods have been implemented in the
new open-source R package cases which allows to reproduce all simulation
results. Based on our numerical results, we conclude that the parametric
approaches (maxT, Bonferroni) are easy to apply but can have inflated type I
error rates for small sample sizes. The two investigated Bootstrap procedures,
in particular the so-called pairs Bootstrap, allow for a family-wise error
rate control in finite samples and in addition have a competitive statistical
power.

_K_ eywords diagnosis $\cdot$ medical testing $\cdot$ multiple testing $\cdot$
model selection $\cdot$ prediction $\cdot$ prognosis

##  1 Introduction

The aim of diagnostic and prognostic studies is generally to differentiate
between two groups, e.g. the diseased from the non-diseased or those with good
outcome from those with poor outcome. This differentiation can be based on a
single clinical parameter or a combination of parameters. The methods and
results presented here are independent of whether the goal is diagnosis or
prognosis and whether a single parameter or a combination is evaluated.
Furthermore, this work also covers the scenario that several investigated
diagnostic procedures are defined by different (machine-learned) prediction
models which are based on the same input data, e.g. medical images. For better
readability, the term diagnosis, diagnostic study and diagnostic test is used
in the following, but prognosis, prognosis study and prognostic marker are
meant at the same time.

In order to assess the accuracy of a new diagnostic test, the so-called index
test, the result is compared with the true condition, which is assessed using
the gold standard or reference standard. The recommended co-primary endpoints
are sensitivity as the proportion correctly diagnosed as diseased and
specificity as the proportion correctly diagnosed as non-diseased. If a
standard test exists, the index test should be compared with it, preferably in
the within-subject design (all tests in all individuals). However, if there is
no established comparator, the aim is to demonstrate a pre-specified minimum
sensitivity and specificity, the so-called minimally acceptable criteria (MAC)
(European Medicines Agency, CHMP, 2010; Korevaar ., 2019).

Since the study is only considered a success if both hypotheses regarding
sensitivity and specificity can be rejected, no correction is required for the
multiplicity of the two endpoints (European Medicines Agency, CHMP, 2010).
However, multiplicity problems may occur elsewhere in diagnostic studies,
especially if the aim is to evaluate different markers in parallel and select
the best diagnostic test or to determine the optimal cut-off value for a
single marker. Usually, a test or cut-off value is selected in a first step
data-driven and then validated in a new study. However, it is known that data-
driven selection may lead to an overestimation of the diagnostic accuracy and
that the results of the selection study often cannot be replicated in the
validation study (Leeflang, 2008). The same phenomenon, sometimes referred to
as optimization bias or selection-induced bias, is well known to also exist in
bioinformatics and predictive modelling (Boulesteix Strobl, 2009; Jelizarow .,
2010).

Therefore, Westphal Brannath (2020) have proposed a framework for the
evaluation of multiple prediction models in which not the (apparently) single-
best model is chosen and subsequently validated, but rather a set of promising
models from which the best is selected in the evaluation study. Westphal .
(2022) have transferred the framework to diagnostic accuracy studies with co-
primary endpoints sensitivity and specificity. In the same way, the framework
can also be used to determine the optimal cut-off value for a single biomarker
after choosing a range of promising cut points in a prior development study.
Regardless of the context, this framework offers increased flexibility and and
can increase statistical power, i.e. the probability to correctly demonstrate
a sufficiently high diagnostic accuracy for one of the selected tests.

However, it is important to correct for multiplicity in the validation study
to avoid inflation of the type-one error. The correction approaches can be
divided into single-step and stepwise procedures. In contrast to stepwise
procedures, single-step procedures have the advantage that associated
simultaneous confidence intervals can be constructed (Strassburger Bretz,
2008). This is relevant in diagnostic accuracy studies as confidence intervals
are an important indicator of estimation uncertainty.

The simplest single-step correction is the Bonferroni adjustment, in which the
adjusted significance level is equal to the global significance level, say
$\alpha=0.05$, divided by the number of tests. This approach leads to
conservative results when the diagnostic test results are positively
correlated. This is for instance the case if the same biomarker or risk score
is dichotomized at slightly different cut points. More elaborate multiple
comparison procedures considered in this work aim to exploit the correlation
structure and thereby increase statistical power. Existing approaches include
the maxT-approach (Hothorn ., 2008) utilized by Westphal Brannath (2020). This
method is based on a multivariate normal approximation which leads to less
conservative decisions in light of positive correlations. Nonparametric
methods for diagnostic test evaluation were developed by Konietschke . (2012)
and Zapf . (2015) transferred this approach to early diagnostic studies with
the area under the receiver operating characteristic (ROC) curve, the so-
called AUC as primary outcome measure. Rudolph (2017) in turn transferred the
approaches from the AUC to sensitivity and specificity as co-primary
endpoints.

The main contribution of this work is the adaptation of several multiple
comparison procedures to diagnostic accuracy studies with multiple index tests
with sensitivity and specificity as co-primary endpoints. The results of an
extensive simulation study are presented to allow a systematic, neutral and
reproducible comparison of relevant statistical properties. Finally, we
introduce a new open-source R package
cases111https://github.com/maxwestphal/cases, last accessed . which provides
convenient implementations for all investigated methods.

This work is structured as follows. Section 2 presents a motivational real-
world data example. Different multiple comparison procedures and the design of
the simulation study are introduced in Section 3. Section 4 then presents the
results of the simulation study and the example data set. A discussion of the
results and the conclusion can be found in Section 5.

##  2 Motivating example: Breast cancer diagnosis

For the motivating example, we utilize the "Breast Cancer Wisconsin
(Diagnostic) Data
Set"222https://archive.ics.uci.edu/ml/datasets/breast+cancer+wisconsin+(diagnostic)
(last assessed ). In total, the dataset consists of 569 observations of
digitized image of a fine needle aspirate of a breast masses. Hereby,
$n_{1}=212$ observations are breast cancer cases and $n_{0}=357$ are controls
(fibrocystic breast masses). Thirty real-valued features that were computed
from the image data which describe characteristics of the cell nuclei present
in the image. A detailed description of the dataset and feature extraction
methodology is provided by Street . (1993); Wolberg . (1994, 1995);
Mangasarian . (1995). Two (independent) example scenarios are described below
and analysed in Section 4.2. For both scenarios, we assume that a high
sensitivity is deemed more important compared to a high specificity because
false negatives have more serious consequences than false positives.

###  2.1 Scenario A: Biomarker assessment

In this first example scenario, our goal is the assessment of the diagnostic
accuracy of multiple biomarker candidates in a single study. In this example,
three of the thirty available features, have been selected before the study
based on prior data and knowledge. The selected biomarker candidates are the
most extreme area, compactness and concavity of any of the ceil nuclei. To
define a diagnostic test based on a continuous biomarker, a threshold needs to
be specified in addition to allow categorization into diseased and healthy
subjects. However, the optimal threshold can usually only be specified with
uncertainty before the study. In our example, we specify five threshold
candidates for each biomarker, roughly corresponding to the 30%, 40%, 50%, 60%
and 70% quantiles of each biomarker distribution. In this scenario, our
framework will be used to investigate the diagnostic accuracy of the resulting
15 combinations of markers (3) and thresholds (5 per marker) simultaneously
while adjusting for the resulting multiplicity.

###  2.2 Scenario B: Risk model evaluation

In modern medical applications, individual predictive features are commonly
combined to risk prediction models. In this second example scenario, we
evaluate several candidate models simultaneously. We split the dataset into
379 and 190 observations for model learning and evaluation, respectively, to
emulate a prospective evaluation study after model development. In the
development phase, five variants of the elastic net algorithm are employed,
corresponding to five different values of the penalty mixing hyperparameter
$\alpha\in\\{0,0.25,0.5,0.75,1\\}$ (Zou Hastie, 2005; Friedman ., 2010). For
each of the five variants, the optimal penalty strength $\lambda$ is found via
10-fold-cross-validation. In the subsequent evaluation phase, the five
optimized models ought to be assessed simultaneously. Because each model
provides a predicted probability of cancer, the question arises which
probability threshold should be used in practice to make classifications. As a
high sensitivity is targeted, the thresholds 10%, 20%, 30%, 40%, 50% will be
evaluated on the remaining ’prospective’ evaluation data. In effect, in this
scenario, our framework will be used to investigate the diagnostic accuracy of
the resulting 25 combinations of risk models (5) and probability thresholds (5
per model) simultaneously while adjusting for the resulting multiplicity.

##  3 Methods

In the following, the statistical model and the hypotheses are presented
(3.1). Various approaches which allow for a multiplicity adjusted analysis are
then described (3.2 and 3.3). Finally, the design of the simulation study is
outlined (3.4).

###  3.1 Statistical model and hypotheses

A sample of $n$ individuals is assumed, divided into $n_{0}$ non-diseased and
$n_{1}$ diseased (with $n=n_{0}+n_{1}$). We consider the within-subject
design, i.e. for each person $i=1,\ldots,n$, the reference standard and $m$
index tests are applied. The comparison of each test $T_{j}$, $j=1,\ldots,m$,
and reference standard $D$ results in a true sensitivity
$\operatorname{se}_{j}=P(T_{j}=1|D=1)$ and specificity
$\operatorname{sp}_{j}=P(T_{j}=0|D=0)$. Hereby, $T_{j}\in\\{0,1\\}$ indicates
test-positive (1) and test-negative (0) and $D\in\\{0,1\\}$ indicates diseased
(1) and non-diseased (0) individuals, respectively. As stated in the
introduction, the different index tests $T_{j}$ may be based on a single
clinical parameter or biomarker which is dichotomized based on different cut
points. On the other hand, each $T_{j}$ may also be the result of a different
complex prediction model based on high-dimensional input data (e.g. medical
images). A mixture of simple and complex models is also possible.
We consider the two alternative hypotheses that true sensitivity and
specificity are above a pre-specified minimum sensitivity and specificity,
which are denoted by $\operatorname{se}_{0}$ and $\operatorname{sp}_{0}$
(Korevaar ., 2019). This leads to the individual null hypotheses

$\displaystyle
H_{0,j}^{\operatorname{se}}:\operatorname{se}_{j}\leq\operatorname{se}_{0},\quad
H_{0,j}^{\operatorname{sp}}:\operatorname{sp}_{j}\leq\operatorname{sp}_{0}$
(1)

for each test $T_{j}$. These two hypotheses are connected according to the
intersection-union principle to the combined null hypothesis

$\displaystyle H_{0,j}:\ H_{0,j}^{\operatorname{se}}\ \cup\
H_{0,j}^{\operatorname{sp}}$ (2)

per index test $T_{j}$, $j=1,\ldots,m$. The global null hypothesis for our
multiple testing problem then reads as

$\displaystyle H_{0}:\
\bigcap_{j=1}^{m}H_{0,j}=\bigcap_{j=1}^{m}\left\\{H_{0,j}^{\operatorname{se}}\
\cup\ H_{0,j}^{\operatorname{sp}}\right\\}$ (3)

Usual (maximum-likelihood) estimates of sensitivity ($\widehat{se}_{j}$) and
specificity ($\widehat{sp}_{j}$) are calculated as

$\displaystyle\widehat{se}_{j}=\frac{n_{j,11}}{n_{1}}\quad\mbox{and}\quad\widehat{sp}_{j}=\frac{n_{j,00}}{n_{0}}$
(4)

with $n_{j,11}$ as number of diseased individuals with a positive test result
and $n_{j,00}$ as number of non-diseased individuals with a negative test
result, each for test $j$. In the cases package, the estimates (4) are
actually slightly shrunk towards $0.5$ by default to prevent singular
(co)variance estimates Westphal . (2022). The decision to retain or reject any
null hypothesis can be made by means of a multiple comparison procedure as
discussed in the next two sections.

###  3.2 Multiple comparison procedures

As a starting point for the data analysis we define usual Wald test statistics
for the $j$-th index test by

$\displaystyle
Z^{\operatorname{se}}_{j}=\frac{\widehat{\operatorname{se}}_{j}-\operatorname{se}_{0}}{\sqrt{\widehat{\operatorname{se}}_{j}(1-\widehat{\operatorname{se}}_{j})/n_{1}}}\quad\mbox{and}\quad
Z^{\operatorname{sp}}_{j}=\frac{\widehat{\operatorname{sp}}_{j}-\operatorname{sp}_{0}}{\sqrt{\widehat{\operatorname{sp}}_{j}(1-\widehat{\operatorname{sp}}_{j})/n_{0}}}.$
(5)

A level $\alpha$ test for the combined hypothesis $H_{j}$ (without adjustment
for multiplicity) can easily be obtained. For this, both individual test
statistics need to surpass an appropriate critical value $c_{\alpha}$
(European Medicines Agency, CHMP, 2010; Korevaar ., 2019). In the simplest
case, $c_{\alpha}=z_{1-\alpha}$ is the $1-\alpha$ quantile of the standard
normal distribution. With this choice, we can define the rejection criterion

$\displaystyle\varphi_{j}=1\ \Longleftrightarrow\
Z_{j}=\min(Z^{\operatorname{se}}_{j},Z^{\operatorname{sp}}_{j})>c_{\alpha}\
\Longleftrightarrow\ Z^{\operatorname{se}}_{j}>c_{\alpha}\wedge
Z^{\operatorname{sp}}_{j}>c_{\alpha}.$ (6)

This approach gives approximate control of the type I error for $H_{0,j}$
under arbitrary parameter configurations
$(\operatorname{se}_{j},\operatorname{sp}_{j})$. Due to asymptotic nature of
the procedure, the type I error is often inflated for small sample sizes and
proportions close to 0 or 1 (Newcombe, 1998). Most importantly, even in the
asymptotic case, it does not provide control of the family-wise error rate
(FWER) for $H_{0}$ which is defined as probability to obtain at least one
false positive rejection $\varphi_{j}=1$ of one of the true null hypotheses
$H_{0,j}$, $j\in 1,\ldots,m$.

Various possibilities exist to allow for control of the FWER of which several
are investigated in this work. All methods are based on existing approaches
which however need to be adapted to the special structure of the hypothesis
problem (3). The required adaptation is similar for all methods and implies a
higher power compared to a native application. This is illustrated in detail
based on corresponding comparison regions in Section 3.3. In the following, we
briefly describe all investigated methods.

  * •

no adjustment: As outlined above, this naive approach is based on (6) and
$c_{\alpha}=z_{1-\alpha}$ chosen to be the $1-\alpha$ quantile of the standard
normal distribution.

  * •

Bonferroni: This well-known multiplicity correction can also be described by
(6) but $c_{\alpha}=z_{1-\alpha/m}$ now corresponds to the standard normal
quantile for an adjusted local significance level $\alpha^{*}=\alpha/m$

  * •

maxT: This asymptotic parametric approach is based on a multivariate normal
approximation of the relevant vector of test statistics (Westphal ., 2022).
The critical value $c_{\alpha}$ in (6) is calculated as an equicoordinate
quantile of the multivariate normal distribution with expectation $\mathbf{0}$
and estimated correlation matrix $\mathbf{\widehat{R}}$. Positively correlated
diagnostic test decisions imply a smaller $c_{\alpha}$ and thus less
conservative test results.

  * •

pairs Bootstrap: A nonparametric approach based on (6) whereby $c_{\alpha}$ is
chosen to be the empirical $1-\alpha$ quantile of a bootstrap sample of the
maximum test statistic
$Z=\max_{j}(Z_{j})=\max_{j}\min(Z^{\operatorname{se}}_{j},Z^{\operatorname{sp}}_{j})$.
For that matter, bootstrap samples of the original data are drawn in a
’paired’ fashion such that the original correlation structure of the problem
is replicated. A general introduction to this and the next bootstrap approach
in the context of regression models is given by Flachaire (2005).

  * •

wild Bootstrap: This approach is similar to the pairs bootstrap but the
mechanism to draw new bootstrap samples is different (Flachaire, 2005; Zapf .,
2015). This approach is investigated although it is not designed for binary
data as bootstrap redraws are not necessarily binary.

  * •

mBeta: A Bayesian approach whereby two independent multivariate beta-binomial
models are fitted for diseased and healthy subgroups (Westphal, 2019). A
rather particular loss function needs to be employed to achieve (frequentist)
control of the FWER relevant for hypothesis problem (3).

All methods are implemented in the new R package cases. Implementation details
can be checked in the open source code and corresponding documentation.
Besides the Bayesian approach, all methods are capable to provide adjusted
p-values for decision making.

The above-mentioned hypotheses (2) can be adapted to the case that all index
tests $T_{j}$ are compared to a common comparator $T_{0}$, i.e. established
diagnostic test with unknown diagnostic accuracy, instead of fixed performance
thresholds $\operatorname{se}_{0}$ and $\operatorname{sp}_{0}$. All
statistical test procedures listed above are also capable to perform more
general contrast tests and allow to specify non-inferiority or superiority
margins. In the simulation study in Section 3.4, we however only consider the
scenario described in Section 3.1.

###  3.3 Confidence and comparison regions

Besides multiplicity adjusted test decisions and p-values, a (two-dimensional)
uncertainty quantification is also important in diagnostic accuracy studies.
Confidence regions are the multivariate counterpart to confidence intervals.
In our case of $m$ index tests, each with unknown sensitivity and specificity,
we are interested in $2m$-dimensional confidence regions
$\bm{C}=\bm{C}_{1-\alpha}$ with coverage probability $1-\alpha$, i.e. we
require

$\displaystyle\forall\bm{\theta}=(\boldsymbol{\operatorname{se}},\boldsymbol{\operatorname{sp}})=(\operatorname{se}_{1},\ldots,\operatorname{se}_{m},\operatorname{sp}_{1},\ldots,\operatorname{sp}_{m})\in[0,1]^{2m}:\quad\mathbb{P}\left(\bm{\theta}\in\bm{C}\right)\geq
1-\alpha.$ (7)

We focus our attention on rectangular confidence regions $\bm{C}$ which are
the Cartesian product of test specific confidence regions
$\bm{C}_{j}=C_{j}^{\operatorname{se}}\times C_{j}^{\operatorname{sp}}$ with
$C_{j}^{\operatorname{se}}\subset[0,1]$ and
$C_{j}^{\operatorname{sp}}\subset[0,1]$. For simplicity, we only consider one-
sided intervals/regions in the following, with all upper limits equal to one.

The duality of confidence intervals and statistical tests for a single
parameter is well-known. Similarly, we might expect that defining a
statistical test by rejecting $H_{j}$ when both $\operatorname{se}_{0}\notin
C_{j}^{\operatorname{se}}$ and $\operatorname{sp}_{0}\notin
C_{j}^{\operatorname{sp}}$ from (2) gives us valid, multiplicity adjusted test
decisions. While this approach indeed allows (approximate) control of the
FWER, we will illustrate in the following that this procedure is conservative
and can easily be improved.

Eckert Vach (2020) introduce the concept of comparison regions to display the
uncertainty in sensitivity and specificity of a diagnostic procedure. We
define the region of interest
$\bm{R}=\\{(\operatorname{se},\operatorname{sp})\in[0,1]^{2}:\operatorname{se}>\operatorname{se}_{0}\wedge\operatorname{sp}>\operatorname{sp}_{0}\\}$
as the complement of $H_{0,j}$ in $[0,1]^{2}$. A comparison (or decision)
region for a single index test is a region $\bm{D}_{j}$ such that defining a
statistical test $\varphi_{j}$ via

$\displaystyle\varphi_{j}=1\ \Longleftrightarrow\ \bm{D}_{j}\subset\bm{R}$ (8)

allows control of the type I error rate. In other words, a statistical test
decision based on the comparison region indicates a significant result if
$\bm{D}_{j}$ is completely contained in $\bm{R}$. In contrast to Eckert Vach
(2020), we focus on rectangular regions of interest in this work and in
addition introduce the concept of multiplicity adjusted comparison regions.

The simplest way to illustrate the difference between confidence and
comparison regions is the Bonferroni adjustment. For a single index test
($j=m=1$), consider the region

$\displaystyle\bm{A}_{j,\alpha^{*}}=\left(\widehat{\operatorname{se}}_{j}-c_{\alpha^{*}}\sqrt{\frac{\widehat{\operatorname{se}}_{j}(1-\widehat{\operatorname{se}}_{j})}{n_{1}}},1\right]\times\left(\widehat{\operatorname{sp}}_{j}-c_{\alpha^{*}}\sqrt{\frac{\widehat{\operatorname{sp}}_{j}(1-\widehat{\operatorname{sp}}_{j})}{n_{0}}},1\right],$
(9)

with $c_{\alpha^{*}}=z_{1-\alpha^{*}}$ being the $1-\alpha^{*}$ standard
normal quantile. Then, $\bm{C}_{j,\alpha}=\bm{A}_{j,\alpha/2}$
($\alpha^{*}=\alpha/2$) defines an (asymptotic) confidence region for
$\bm{\theta}_{j}=(\operatorname{se}_{j},\operatorname{sp}_{j})$. An adjustment
for multiplicity is needed here, as both, sensitivity and specificity shall be
covered by $\bm{C}$ with high probability, compare (7). On the other hand, the
comparison region, as the statistical test for the co-primary endpoint
problem, does not require an adjustment for $\alpha$ and hence
$\bm{D}_{j,\alpha}=\bm{A}_{j,\alpha}$ ($\alpha^{*}=\alpha$).

This directly extends to multiplicity adjusted confidence and comparison
regions when $m>1$ index tests are simultaneously evaluated. Here, the
Bonferroni adjustment is $\alpha^{*}=\alpha/(2m)$ for the confidence region,
as $2m$ parameters shall be covered. The adjustment for the corresponding
comparison region however is only $\alpha^{*}=\alpha/m$. All statistical
methods introduced in Section 3 allow to calculate multiplicity adjusted
confidence and comparison regions. For all procedures, one will find that
$\bm{D}_{\alpha}\subset\bm{C}_{\alpha}$ and thus decisions based on confidence
regions will not be dual to test decisions but rather more conservative.

Figure 1 illustrates this difference for a synthetic data set with $n_{1}=30$
cases and $n_{0}=90$ controls and $m=4$ index tests with varying accuracy. The
region of interest $\bm{R}$ is displayed in the upper right-hand corner in
green. In the left subfigure, multiplicity adjusted comparison regions by the
maxT-approach are indicated by solid lines around each point estimate
$(\widehat{\operatorname{se}}_{j},\widehat{\operatorname{sp}}_{j})$. The study
conclusion in this case would be a rejection of the null $H_{0,j}$ only one of
the four index tests ($j=1$, solid/blue lines) as $\bm{D}_{1}\subset\bm{R}$
but $\bm{D}_{j}\not\subset\bm{R}$ for all other tests $j=2,3,4$ (dashed/orange
lines). The right subfigure reveals that a decision based on the corresponding
(maxT-adjusted) confidence regions would have not resulted in any rejection of
the null hypothesis as $\bm{C}_{j}\not\subset\bm{R}$ for all $j=1,2,3,4$.

Figure 1: Exemplary analysis of synthetic example with four index tests.
Region of interest ($\operatorname{se}_{0}=0.8$, $\operatorname{sp}_{0}=0.7$)
is displayed in green. Left: comparison regions. Right: confidence regions.
Solid/blue lines imply a rejected null hypothesis; dashed/orange lines imply a
non-rejected null hypothesis.

###  3.4 Simulation study

We perform a simulation study to evaluate and compare the multiple comparison
procedures described in Section 3.2. Our primary goal is a comparison with
regards to family-wise error rate and statistical power. The (disjunctive)
statistical power is the probability to obtain at least one rejection of a
false null hypothesis $H_{0,j}$. For that matter, two different data-
generating mechanisms (’LFC’, ’Biomarker’) are investigated and outlined in
the following. The key features of the simulation study are described in Table
1 following the ADEMP framework (Morris ., 2019; Boulesteix ., 2020).

Feature |  Description
---|---
Aims |  Comparison of different multiple comparison procedures (MCPs)
Data generating mechanisms |  Different scenarios (w.r.t. sample size, control-to-case ratio, correlation structure, …) in two distinct settings (’LFC’: Section 3.4.1, ’Biomarker’: Section 3.4.2)
Methods of
analysis |  Multiple comparison procedures for hypothesis problem (3), described in Section 3.2
Performance measures |  Family-wise error rate (FWER) and (disjunctive) power
Number of
repetitions |  $n_{sim}=10,000$ per scenario
Table 1: Key features of simulation study

For both simulation settings (compare Section 3.4.1 and 3.4.2), many different
scenarios are considered which are described by a number of design parameters
(setting, sample size, prevalence, correlation structure, …). For each
scenario, $n_{sim}=10,000$ synthetic datasets were generated. The simulation
study is conducted such that all methods are applied to the exact same
synthetic datasets. The standard (Monte-Carlo) error for estimated proportions
$p$ such as the FWER or statistical power is $(p(1-p)/n_{sim})^{1/2}$ which is
bounded by $0.005$ for $n_{sim}=10,000$. In all simulation scenarios, we
investigated sampling with a fixed sample sizes $n_{0}$, $n_{1}$ which
corresponds to a simple case-control within-subjects study design.

All numerical experiments presented in this work are reproducible by means of
the new R package cases. Instruction for that matter are provided via an
additional simulation repository which also contains the results of additional
sensitivity analyses333https://maxwestphal.github.io/cases_simstudy/, last
accessed .. As the source code for all statistical methods is available in the
cases package, this also serves as a documentation of implementation-related
details. To reduce the computational demand of the simulation study, we
consider only the comparison of $m$ candidate tests to fixed performance
criteria $\operatorname{se}_{0}=\operatorname{sp}_{0}$. The cases package
furthermore supports comparison of $m$ candidate tests to the unknown
sensitivity and specificity of a comparator and other relevant contrast tests.
The simulations were conducted in R (version 4.2.1) with help of the
batchtools444https://CRAN.R-project.org/package=batchtools, last accessed .
package.

####  3.4.1 LFC setting

In the first setting, a single synthetic data set consists of two separate
samples for diseased and healthy. For that matter, two different multivariate
Binomial models are employed to generate binary data matrices $\bm{Q}^{1}$ and
$\bm{Q}^{0}$. These matrices have entries

$\displaystyle Q_{ij}^{g}=\mathds{1}(T_{ij}=D_{i}),\quad i=1,\ldots,n,\quad
j=1,\ldots,m,\quad g=0,1,$ (10)

whereby $Q_{ij}^{g}$ indicates if the $j$-th diagnostic test (data-based
decision rule) $T_{j}$ under consideration correctly predicted the $i$-th
disease status $D_{i}$ in subgroup $g\in\\{0,1\\}$. The multivariate binomial
distribution in general has $2^{m}$ parameters but reduced representations do
exist which can be specified by the mean vector and correlation matrix. We
utilize the R package bindata555https://CRAN.R-project.org/package=bindata,
last accessed . to generate the data $Q^{\operatorname{se}}$,
$Q^{\operatorname{sp}}$ with given first moments
$\boldsymbol{\operatorname{se}}$, $\boldsymbol{\operatorname{sp}}$,
respectively (Leisch ., 1998).

In this scenario, we focus on least-favorable parameter configurations (LFC)
which are defined as parameters which maximize the FWER. As discussed by
Westphal Brannath (2020), an LFC in this case is defined by mean vectors
$\boldsymbol{\operatorname{se}}$, $\boldsymbol{\operatorname{sp}}$ with
entries

$\displaystyle\operatorname{se}_{j}=\begin{cases}\operatorname{se}_{0},&b_{j}=1\\\
1,&b_{j}=0\end{cases},\quad\operatorname{sp}_{j}=\begin{cases}\operatorname{sp}_{0},&b_{j}=0\\\
1,&b_{j}=1\end{cases},$ (11)

whereby $\operatorname{se}_{0}$, $\operatorname{sp}_{0}$ are the parameter
thresholds which define the hypotheses (1) and
$\bm{b}=(b_{1},\ldots,b_{m})\in\\{0,1\\}^{m}$ is a fixed binary vector. For
each index $j$ we thus have either $\operatorname{se}_{j}$ or
$\operatorname{sp}_{j}$ exactly at the boundary of the null hypothesis and the
other parameter equal to one. For each of the two endpoints, a correlation
matrix is specified in addition. To limit computational resources, we
restricted the attention to equicorrelation between non-degenerated
parameters, i.e. $\rho^{se}_{jj^{\prime}}=\rho^{\operatorname{se}}$ for all
indices $j\neq j^{\prime}$ with $\operatorname{se}_{j}\neq 1$ and
$\operatorname{se}_{j^{\prime}}\neq 1$.

####  3.4.2 Biomarker setting

The least-favorable parameter (LFC) setting discussed in the last section
allows to assess worst case error rates. However, LFCs are rarely
representative of real-world situations (Westphal ., 2022). The goal of this
second simulation study is thus to evaluate the multiple comparison procedures
under more realistic parameter configurations.

To generate a single synthetic data set, we start out by simulating $l$
continuous biomarkers from a multivariate binormal ROC model (Pepe, 2003,
Section 4.4). This means, that we consider multivariate markers
$\bm{V}=(V_{1},\ldots,V_{l})$ such that

$\displaystyle\bm{V}\,|\,D=1\sim\mathcal{N}(\bm{\mu}^{1},\bm{R}^{1}),\quad\bm{V}\,|\,D=0\sim\mathcal{N}(\bm{0},\bm{R}^{0}).$
(12)

Hereby, $\mu^{1}_{k}$, is the mean of marker $k$ in the diseased ($D=1$)
population. For simplicity, we assume a mean $\mu^{0}_{k}=0$ for all
$k=1,\ldots,l$ in the healthy population ($D=0$) and a variance of one in both
populations. If we aim for a certain AUC of marker $k$, this can be achieved
by solving

$\displaystyle\operatorname{AUC}_{k}=\Phi\left(\frac{\mu_{k}^{1}}{\sqrt{2}}\right)$
(13)

for $\mu^{1}_{k}$ which has the solution
$\mu_{k}^{1*}=\sqrt{2}\Phi^{-1}(\operatorname{AUC}_{k})$ (Pepe, 2003, Section
4.4.2). Hereby, $\Phi$ is the cumulative distribution function of the standard
normal distribution.

These continuous scores $V_{k}$ are split at different cut points $c_{j}$ to
obtain an overall list of diagnostic tests
$T_{j}=\mathds{1}(V_{k_{j}}>c_{j})$, $j=1,\ldots,m$ with $m\geq l$. The
resulting decision rules have true parameters

$\displaystyle\operatorname{se}_{j}=\Phi\left(\mu^{1}_{k_{j}}-c_{j}\right),\quad\operatorname{sp}_{j}=1-\Phi\left(-c_{j}\right)$
(14)

depending on the cut point $c_{j}$. Alternatively, this process can also be
identified as thresholding risk model scores to obtain different prediction
models. The correlation matrices $\bm{R}^{0},\bm{R}^{1}$ in (12) have a simple
structure (e.g. equicorrelation) in our simulation study. Together with the
chosen cut points, this induces a certain correlation structure for the actual
binary decision rules. Naturally, two (close) nearby cut points, induce
(highly) similar predictions and thus a (highly) positive correlation between
estimates.

##  4 Results

###  4.1 Simulation study

####  4.1.1 LFC setting

The results from the LFC (least-favorable configuration) setting described in
Section 3.4.1 are displayed in Figure 2. On the left, the FWER is shown
depending on the total sample size $n=n_{1}+n_{0}$ for a control-to-case ratio
of $3:1$ and $m=10$ index tests. The parameter values are set to an LFC with
$\operatorname{se}_{0}=\operatorname{sp}_{0}=0.8$, compare Section 3.4.1. The
red dashed vertical line shows the (one-sided) target significance level of
$\alpha=0.025$.

First, applying no multiplicity adjustment at all, clearly leads to a vastly
increased FWER of $20\%$ and more compared to the target significance level of
$\alpha=0.025$. Second, the asymptotic methods (maxT, Bonferroni) are capable
of controlling the FWER for large $n$ but fail to do so for small $n$. For
$n=100$ both methods show an increased FWER of up to 10%, declining to around
$5\%$ for $n\geq 800$. Third, the Boostrap approaches (pairs, wild) and the
Bayesian mBeta approach all control the FWER for all but the lowest sample
size of $n=100$.

The right part of Figure 2 shows the (disjunctive) power for the same data
instances after dropping the minimal acceptance criteria
$\operatorname{se}_{0},\operatorname{sp}_{0}$ both to $0.75$. The ordering of
methods stays the same as for the FWER analysis, as expected. The gap between
mBeta and Bootstrap approaches is somewhat more pronounced than before.

Further analyses (not shown visually here) indicate, that these findings for
FWER and power remain qualitatively similar when only $m=5$ index tests are
considered are when the correlation structure between diagnostic test result
is changed. When the control-to-cases ratio is changed to $10:1$, all curves
are essentially shifted. To obtain a similar power as before, a higher total
sample size is required due to the dependence on $\min(n_{1},n_{0})$. As
expected, when $\operatorname{se}_{0}$ is changed from 0.8 to 0.9, the FWER is
increased for finite samples but the qualitative observations regarding the
asymptotic behaviour still persist.

Figure 2: Simulation results for the ’LFC’ setting. Left: FWER, Right: Power.
The horizontal axis shows the total sample size (cases and controls
$n=n_{1}+n_{0}$) in both subplots.

Figure 3: Simulation results for the ’Biomarker’ setting. Left: FWER, Right:
Power. The horizontal axis shows the total sample size (cases and controls
$n=n_{1}+n_{0}$) in both subplots.

####  4.1.2 Biomarker setting

Figure 3 shows results from the simulation in the more realistic biomarker
setting described in Section 3.4.2. In contrast to the worst case assessment
in the last section, the FWER is below $0.005$ for all methods and nearly all
sample sizes and as such far below the target significance level of
$\alpha=0.025$. This behaviour is expected based on previous work (Westphal
Brannath, 2020).

Regarding statistical power the ordering of methods is mostly similar to the
situation in the LFC setting. One important difference is the much better
performance of both Bootstrap approaches which seem to adapt much better to
the underlying generative distribution in this situation.

###  4.2 Analysis of motivating example

To conclude, we apply our methodology to real-world data and interpret the
results. For that matter, the pairs bootstrap approach is applied in the two
motivating example scenarios introduced in Section 2. To formalize our
requirement that a high sensitivity is prioritized, we define the minimal
acceptance criteria as $\operatorname{se}_{0}=0.9$ and
$\operatorname{sp}_{0}=0.7$. The entire analysis can be reproduced by
following the corresponding vignette of the new R package cases.

####  4.2.1 Scenario A: Biomarker assessment

The most promising of the 15 investigated classification rules in terms of
diagnostic accuracy on the evaluation data (212 cases, 357 controls) is the
maximal area of any present cell nucleus with a threshold of $700\mu m^{2}$
applied for categorization into cases ($>700\mu m^{2}$) and controls ($\leq
700\mu m^{2}$). This rule achieves an empirical sensitivity of $96.0\%$ and an
empirical specificity $80.6\%$. The multiplicity adjusted lower comparison
bounds are $92.1\%$ and $74.6\%$, respectively. As both lower bounds are
larger than their respective accepetance criteria, the co-primary null
hypothesis can be rejected for this test candidate. This is visualized in
Figure 4 (left). None of the other comparison regions is completely contained
in the region of interest corresponding to the alternative hypothesis. In
effect, the null hypothesis cannot be rejected for any other test candidate
even though individual sensitivity and specificity estimates of other
candidates are larger than those reported above.

####  4.2.2 Scenario B: Risk model evaluation

In total, 25 risk prediction models were assessed on the evaluation data (71
cases, 119 controls). For four models, the null hypothesis
$\operatorname{se}\leq 0.9\vee\operatorname{sp}\leq 0.7$ can be rejected as
these models have lower comparison bounds for sensitivity and specificity that
greater than the corresponding acceptance criteria. Empirical sensitivities of
these models lie between $99.3\%$ and $97.9\%$ and specificities between
$88.8\%$ and $85.4\%$. This is visualized in Figure 4 (right). These four
classifiers all result from thresholding different model risk scores at the
lowest probability threshold of $0.10$. Because all four models are deemed
acceptable regarding their discriminatory performance, other criteria may
influence the final model decision. For instance, the most parsimonious model
can be chosen to increase interpretability. In this example, the lasso fit
(elastic net mixing parameter $\alpha=1$) results in only 11 non-zero
regression coefficients and is much sparser as the three competitors (with
$\alpha<1$) which have 18, 19 and 23 non-zero coefficients, respectively.

Figure 4: Results from the analysis of the two breast cancer diagnosis example
scenarios. Left: biomarker assessment (scenario A). Right: risk model
evaluation (scenario B). Solid/blue lines imply a rejected null hypothesis;
dashed/orange lines imply a non-rejected null hypothesis.

##  5 Discussion

In this work, we investigated statistical inference methods for diagnostic
accuracy studies with multiple index tests and co-primary endpoints
sensitivity and specificity. Multiplicity corrections are relevant in this
context as omitting a suitable correction can otherwise induce overoptimistic
results and inflated error rates. While a control is easily possible using a
traditional (FWER) correction for all $2m$ parameters ($m$ index tests with
unknown sensitivity and specificity), this approach is too conservative for
the co-primary endpoint analysis. Several multiple comparison procedures can
be adapted to the specific testing problem such that this conservative nature
is resolved. This difference was illustrated by contrasting confidence regions
and comparison regions in Section 3.3. Only the latter allow for a two-
dimensional uncertainty quantification which is dual to the statistical test
result.

We conducted an extensive simulation study to compare five multiple comparison
procedures with regards to family-wise error rate (FWER) and statistical
power. All five procedure are capable to control the FWER at least
asymptotically. The Bonferroni adjustment and maxT-approach both need quite
large sample sizes to reach the target significance level. This however
depends crucially on the control-to-cases ratio. A more skewed class
distribution results in higher sample size demand. The Bayesian mBeta approach
and the two Bootstrap (pairs, wild) procedures managed to control the FWER
much better under least-favorable parameter configurations. Based on the
simulation study, we recommend the use of the pairs Bootstrap approach. This
approach yielded good FWER control for small sample sizes and competitive
power in realistic scenarios. The wild bootstrap approach has very similar
performance, but the method is more complex, has more design choices, and was
not originally designed to be used for binary data. For larger sample sizes,
the maxT-approach is a competitive alternative and easier to apply compared to
the bootstrap procedures. The Bonferroni method is by far the easiest method
to apply but too conservative in some situations and as such suboptimal.

In this work, we focused on the family-wise error rate and the statistical
power. In our framework, point estimates are slightly shrunk due to the
implemented minor regularization to avoid singular (co)variance estimates.
However, point estimates were not adjusted for multiplicity. Westphal .
(2022); Westphal Brannath (2020) illustrated how to apply the maxT-approach to
obtain median-conservative point estimators. This generic approach can be also
adapted to the other multiple comparison procedures.

All investigated methods can be adapted to the case that not only two
(diseased and healthy) subpopulations need to be distinguished but more than
two subpopulations, e.g. different disease severities. This capability is
already implemented in the cases package. We expect that the performance will
then crucially depend on the prevalence of the smallest subpopulation. This
should however be confirmed in dedicated simulation studies.

## Conflict of interest

The authors declare that there is no conflict of interest.

## References

  * Boulesteix . (2020) boulesteix_introduction_2020Boulesteix, AL., Groenwold, RH., Abrahamowicz, M., Binder, H., Briel, M., Hornung, R.Sauerbrei, W.  202012\.  Introduction to statistical simulations in health research Introduction to statistical simulations in health research.  BMJ Open1012e039921.  [2021-05-15]https://bmjopen.bmj.com/content/10/12/e039921 Publisher: British Medical Journal Publishing Group Section: Epidemiology  10.1136/bmjopen-2020-039921
  * Boulesteix Strobl (2009) boulesteix_optimal_2009Boulesteix, AL. Strobl, C.  200912\.  Optimal classifier selection and negative bias in error rate estimation: an empirical study on high-dimensional prediction Optimal classifier selection and negative bias in error rate estimation: an empirical study on high-dimensional prediction.  BMC Medical Research Methodology9185.  [2021-05-25]https://doi.org/10.1186/1471-2288-9-85 10.1186/1471-2288-9-85
  * Eckert Vach (2020) eckert_use_2020Eckert, M. Vach, W.  2020\.  On the use of comparison regions in visualizing stochastic uncertainty in some two-parameter estimation problems On the use of comparison regions in visualizing stochastic uncertainty in some two-parameter estimation problems.  Biometrical Journal623598–609.  [2021-05-22]https://onlinelibrary.wiley.com/doi/abs/10.1002/bimj.201800232 _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1002/bimj.201800232  https://doi.org/10.1002/bimj.201800232
  * European Medicines Agency, CHMP (2010) chmp_guideline_2010European Medicines Agency, CHMP.  2010\.  Guideline on clinical evaluation of diagnostic agents. Guideline on clinical evaluation of diagnostic agents.
  * Flachaire (2005) flachaire_bootstrapping_2005Flachaire, E.  200504\.  Bootstrapping heteroskedastic regression models: wild bootstrap vs. pairs bootstrap Bootstrapping heteroskedastic regression models: wild bootstrap vs. pairs bootstrap.  Computational Statistics & Data Analysis492361–376.  [2021-05-23]https://www.sciencedirect.com/science/article/pii/S0167947304001537 10.1016/j.csda.2004.05.018
  * Friedman . (2010) friedman2010Friedman, J., Hastie, T. Tibshirani, R.  2010\.  Regularization paths for generalized linear models via coordinate descent Regularization paths for generalized linear models via coordinate descent.  Journal of statistical software3311.
  * Hothorn . (2008) hothorn_simultaneous_2008Hothorn, T., Bretz, F. Westfall, P.  2008\.  Simultaneous Inference in General Parametric Models Simultaneous Inference in General Parametric Models.  Biometrical Journal503346–363.  [2021-05-22]https://onlinelibrary.wiley.com/doi/abs/10.1002/bimj.200810425 _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1002/bimj.200810425  https://doi.org/10.1002/bimj.200810425
  * Jelizarow . (2010) jelizarow_over-optimism_2010Jelizarow, M., Guillemot, V., Tenenhaus, A., Strimmer, K. Boulesteix, AL.  201008\.  Over-optimism in bioinformatics: an illustration Over-optimism in bioinformatics: an illustration.  Bioinformatics (Oxford, England)26161990–1998.  10.1093/bioinformatics/btq323
  * Konietschke . (2012) konietschke_rank-based_2012Konietschke, F., Hothorn, LA. Brunner, E.  201201\.  Rank-based multiple test procedures and simultaneous confidence intervals Rank-based multiple test procedures and simultaneous confidence intervals.  Electronic Journal of Statistics6none738–759.  [2021-05-22]https://projecteuclid.org/journals/electronic-journal-of-statistics/volume-6/issue-none/Rank-based-multiple-test-procedures-and-simultaneous-confidence-intervals/10.1214/12-EJS691.full Publisher: Institute of Mathematical Statistics and Bernoulli Society  10.1214/12-EJS691
  * Korevaar . (2019) korevaar_targeted_2019Korevaar, DA., Gopalakrishna, G., Cohen, JF. Bossuyt, PM.  201912\.  Targeted test evaluation: a framework for designing diagnostic accuracy studies with clear study hypotheses Targeted test evaluation: a framework for designing diagnostic accuracy studies with clear study hypotheses.  Diagnostic and Prognostic Research3122.  [2021-05-22]https://diagnprognres.biomedcentral.com/articles/10.1186/s41512-019-0069-2 10.1186/s41512-019-0069-2
  * Leeflang (2008) leeflang_systematic_2008Leeflang, MM.  200812\.  Systematic Reviews of Diagnostic Test Accuracy Systematic Reviews of Diagnostic Test Accuracy.  Annals of Internal Medicine14912889.  [2021-05-22]http://annals.org/article.aspx?doi=10.7326/0003-4819-149-12-200812160-00008 10.7326/0003-4819-149-12-200812160-00008
  * Leisch . (1998) leisch_generation_1998Leisch, F., Weingessel, A. Hornik, K.  1998\.  On the generation of correlated artificial binary data On the generation of correlated artificial binary data [Paper].  [2021-05-25]https://epub.wu.ac.at/286/ Issue: 13 Num Pages: 15 Number: 13 Place: Vienna Publisher: SFB Adaptive Information Systems and Modelling in Economics and Management Science, WU Vienna University of Economics and Business
  * Mangasarian . (1995) mangasarian1995Mangasarian, OL., Street, WN. Wolberg, WH.  1995\.  Breast cancer diagnosis and prognosis via linear programming Breast cancer diagnosis and prognosis via linear programming.  Operations Research434570–577.
  * Morris . (2019) morris_using_2019Morris, TP., White, IR. Crowther, MJ.  2019\.  Using simulation studies to evaluate statistical methods Using simulation studies to evaluate statistical methods.  Statistics in Medicine38112074–2102.  [2021-05-15]https://onlinelibrary.wiley.com/doi/abs/10.1002/sim.8086 _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1002/sim.8086  https://doi.org/10.1002/sim.8086
  * Newcombe (1998) newcombe_two-sided_1998Newcombe, RG.  1998\.  Two-sided confidence intervals for the single proportion: comparison of seven methods Two-sided confidence intervals for the single proportion: comparison of seven methods.  Statistics in medicine178857–872.
  * Pepe (2003) pepe_statistical_2003Pepe, MS.  2003\.  The statistical evaluation of medical tests for classification and prediction The statistical evaluation of medical tests for classification and prediction.  Medicine.
  * Rudolph (2017) rudolph_adjusting_2017Rudolph, A.  2017\.  Adjusting for multiplicity in diagnostic studies: A Wild Bootstrap approach for obtaining simultaneous confidence intervals for sensitivity and specificity Adjusting for multiplicity in diagnostic studies: A Wild Bootstrap approach for obtaining simultaneous confidence intervals for sensitivity and specificity .  University of Heidelberg.
  * Strassburger Bretz (2008) strassburger_compatible_2008Strassburger, K. Bretz, F.  2008\.  Compatible simultaneous lower confidence bounds for the Holm procedure and other Bonferroni-based closed tests Compatible simultaneous lower confidence bounds for the Holm procedure and other Bonferroni-based closed tests.  Statistics in Medicine27244914–4927.  [2021-05-22]https://onlinelibrary.wiley.com/doi/abs/10.1002/sim.3338 _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1002/sim.3338  https://doi.org/10.1002/sim.3338
  * Street . (1993) street1993Street, WN., Wolberg, WH. Mangasarian, OL.  1993\.  Nuclear feature extraction for breast tumor diagnosis Nuclear feature extraction for breast tumor diagnosis.  Biomedical image processing and biomedical visualization Biomedical image processing and biomedical visualization ( 1905, 861–870).
  * Westphal (2019) westphal_simultaneous_2019Westphal, M.  2019\.  Simultaneous Inference for Multiple Proportions: A Multivariate Beta-Binomial Model Simultaneous Inference for Multiple Proportions: A Multivariate Beta-Binomial Model.  arXiv preprint arXiv:1911.00098.
  * Westphal Brannath (2020) westphal_evaluation_2020Westphal, M. Brannath, W.  2020\.  Evaluation of multiple prediction models: A novel view on model selection and performance assessment Evaluation of multiple prediction models: A novel view on model selection and performance assessment.  Statistical methods in medical research2961728–1745.  Publisher: SAGE Publications Sage UK: London, England
  * Westphal . (2022) westphal_multiple_2022Westphal, M., Zapf, A. Brannath, W.  2022\.  A multiple testing framework for diagnostic accuracy studies with co-primary endpoints A multiple testing framework for diagnostic accuracy studies with co-primary endpoints.  Statistics in Medicine415891–909.
  * Wolberg . (1995) wolberg1995Wolberg, WH., Street, WN., Heisey, DM. Mangasarian, OL.  1995\.  Computer-derived nuclear features distinguish malignant from benign breast cytology Computer-derived nuclear features distinguish malignant from benign breast cytology.  Human Pathology267792–796.
  * Wolberg . (1994) wolberg1994Wolberg, WH., Street, WN. Mangasarian, OL.  1994\.  Machine learning techniques to diagnose breast cancer from image-processed nuclear features of fine needle aspirates Machine learning techniques to diagnose breast cancer from image-processed nuclear features of fine needle aspirates.  Cancer letters772-3163–171.
  * Zapf . (2015) zapf_wild_2015Zapf, A., Brunner, E. Konietschke, F.  201512\.  A Wild Bootstrap approach for the selection of biomarkers in early diagnostic trials A Wild Bootstrap approach for the selection of biomarkers in early diagnostic trials.  BMC Medical Research Methodology15143.  [2021-05-22]https://bmcmedresmethodol.biomedcentral.com/articles/10.1186/s12874-015-0025-y 10.1186/s12874-015-0025-y
  * Zou Hastie (2005) zou2005Zou, H. Hastie, T.  2005\.  Regularization and variable selection via the elastic net Regularization and variable selection via the elastic net.  Journal of the royal statistical society: series B (statistical methodology)672301–320.


# Analyzing time series activity of Twitter political spambots

Oscar Fontanelli  Centro de Investigaciones Interdisciplinarias en Ciencias y
Humanidades, Universidad Nacional Autónoma de México,
<EMAIL_ADDRESS>Aldo Venegas  Facultad de Ciencias,
Universidad Nacional Autónoma de México  Ricardo Mansilla  Centro de
Investigaciones Interdisciplinarias en Ciencias y Humanidades and Centro
Peninsular en Humanidades y Ciencias Sociales, UNAM

The presence and complexity of political Twitter bots has increased in recent
years, making it a very difficult task to recognize these accounts from real,
human users. We intended to provide an answer to the following question: are
temporal patterns of activity qualitatively different in fake and human
accounts? We collected a large sample of tweets during the post-electoral
conflict in the US in 2020 and performed supervised and non-supervised
statistical learning techniques to quantify the predictive power of time-
series features for human-bot recognition. Our results show that there are no
substantial differences, suggesting that political bots are nowadays very
capable of mimicking human behaviour. This finding reveals the need for novel,
more sophisticated bot-detection techniques.

## Introduction

Digital social networks play an increasingly important role in political
campaigns and conversations all around the world [8, 16, 24]. Rapid diffusion
of information and the possibility of direct interaction between politicians
and citizens allow different political actors to spread their messages in a
way that is efficient, speedy and at a very low cost [2, 9, 11, 25]. In recent
years, the presence of fake or manipulated accounts designed to send spam has
grown considerably within these social networks [22, 23]. In political
contexts, these spambots are created and utilized to massively spread certain
news (usually fake news), opinions, create artificial trends or support
certain persons and organizations, thus contaminating organic conversation
between normal users [1, 19, 20]. The effect they have is that of manipulating
public opinion.

Ever since the emergence of digital social networks there has been an interest
- from public, private and academic points of view - in detecting and
ultimately filtering out these kind of fake accounts [4, 7, 10, 12, 13, 15,
17, 21, 26]. Some of these techniques utilize time series features and they
have been successful in telling human from bots with this information [5, 6,
14], but as algorithms and tools for systematically detecting spambots have
grown popular, so has increased the complexity and the capacity for mimicking
human behaviour that these fake account possess. Nowadays, the problem of
telling between normal and fake accounts requires very sophisticated
techniques and it can be very difficult even with manual, human inspection. In
this context we ask the following question: do human and fake accounts for
political spamming still show qualitative differences in their time activity
patterns?

In this work we analyze political conversations in the social network Twitter
in the context of the post-electoral conflict in the US after the presidential
elections of 2020. By means of supervised and non-supervised statistical
learning techniques we quantify the predictive power of time series features
for distinguishing bots from humans. Our results suggests that time activity
features may be no longer useful for human-bot activity recognition in this
social network.

## Methodology

### Data Collection and Annotation

We utilized the public Twitter API to download 372,768 statuses (tweets,
retweets, quotes or replies) containing either the hashtag $\\#$StopTheSteal
or $\\#$MarchForTrump between November 5, 2020 and January 8, 2021. These
statuses were sent from 208,484 different accounts. Within this time interval
we detected three periods of intense activity (early November, mid November
and early January). Since we are interested in spambots and we want to compute
certain properties of time series, we selected only accounts that posted five
or more statuses within at least one of these three periods, thus remaining
with 9,752 accounts.

We have manually inspected the public profile of these accounts and classified
them into two categories: normal accounts and suspect (of being spambots)
accounts. These classification was done by three independent persons and was
based solely on user metadata from the public profile, not taking into account
any feature of their time series (since we don’t want to construct ant kind of
artificial correlation). This manual inspection yielded 8,782 normal accounts
and 970 suspect accounts.

### Time Series Characterization

For each account $k$ (assume it posted $n_{k}$ statuses) we extracted the
series of time differences between consecutive statuses $\\{\Delta
t_{1},\Delta t_{2},...,\Delta t_{(n-1)_{k}}\\}$. We call this the _time
interval distribution_ of account $k$. Next we computed for each account the
first four moments of this distribution, $m_{i}=\sum(\Delta t_{j})^{i}/n_{k}$,
for $i=1,2,3,4$. We computed two different entropy measures for the time
series: _time entropy_ [18] and _permutation entropy_ [3]. Intuitively, an
account that posts at perfectly regular time intervals (a possible indication
of an automated account) will have an entropy of zero, while high values of
the entropy indicate a more disordered time series.

Feature | Description | Class
---|---|---
n | Number of statuses with the desired hasthag | temporal
m1 | First moment of time interval distribution | temporal
m2 | Second moment of time interval distribution | temporal
m3 | Third moment of time interval distribution | temporal
m4 | Fourth moment of time interval distribution | temporal
time entropy | Time entropy of time interval distribution | temporal
permutation entropy | Permutation entropy of time interval distribution | temporal
followers count | Number of followers | non temporal
friends count | Number of followed accounts | non temporal
statuses count | Total number of statuses since the account was created | non temporal
retweet ratio | Fraction of statuses with the hashtag that are retweets | non temporal
friends-followers ratio | Number of friends / Number of followers | non temporal
age | Age of the account | non temporal
screen name entropy | Shannon entropy of the screen name | non temporal
Table 1: Temporal and non-temporal features we utilized to train supervised
classifiers.

### Supervised classification

We selected two groups of features: those regarding time interval distribution
(temporal features) and a list of characteristics that are typically used to
identify Twitter bots (non temporal features) [4, 7, 10, 15, 21]. We show all
the features we used on table 1. With these features we fitted three models
for our binary classification (normal or suspect account): model 1, using only
temporal features; model 2, using only non-temporal features; model 3, using
all features. Our goal was to see if the predictive power and goodness of fit
measures of these classification models significantly improve when we add
temporal features. We fitted a random forest and a k-nearest neighbors
classifier with models 1, 2 and 3. Hyperparameters of both classifiers were
tuned by a grid search, choosing the best combination according to the area
under the ROC curve criterion. The issue of the imbalanced data was approached
with over and under sampling algorithms.

### Data Clustering

We performed k-means clustering on the Twitter users data with only temporal
features. Our goal was to see whether suspect accounts tend to fall in the
same cluster or group of clusters. If they do, this means that suspect
accounts show similar temporal patterns of activity. We selected the value of
$k$ (number of clusters) by looking for an inflection point in a plot of $k$
against total within sum of squares (the “elbow method”).

| Random Forest | KNN
---|---|---
| Precision | TP rate | Area under ROC | Precision | TP rate | Area under ROC
model 1 | 0.813 | 0.127 | 0.605 | 0.775 | 0.218 | 0.576
model 2 | 0.715 | 0.614 | 0.744 | 0.738 | 0.544 | 0.703
model 3 | 0.730 | 0.614 | 0.758 | 0.758 | 0.519 | 0.740
Table 2: Metrics for models 1, 2 and 3 using random forest and KNN
classifiers.

## Results and discussion

Table 2 shows the main metrics for both classifiers and the three models.
Notice how, because of the imbalance of the data, precision may not be a
reliable metric. See for example results for random forest in model 1:
precision seems to be high, but the extremely low value of the true positive
(TP) rate suggests this is a highly-biased classifier towards the majority
class (normal accounts). Considering this and the area under ROC curve, model
1 (only temporal features) does not seem to be a very good model. Even though
the addition of temporal features increases area under ROC curve (from model 2
to model 3), we see that true positive stays the same or decreases; this means
that model 2 (non temporal features) is actually better than model 3 for
identifying spambots, even though model 3 is a slightly better model overall,
at least for this particular dataset.

Figure 1: Feature importance plot for model 3. Black bars correspond to non-
temporal features, while red ones correspond to temporal features. We also
show ROC curves for models 1, 2 and 3.

Feature importance is a measure in the random forest classifier that measures
the predictive power of each variable. We show in fig. 1 feature importance
for model 3 (all features), along with ROC curves for all three models.
Feature importance was computed using the mean decrease in Gini index. ROC
curves for KNN classifier (not displayed here) show a qualitatively similar
behavior. According to this metric, number of statuses and time entropy are
the most important time-activity features, but all others features of this
class are not very useful for classifying bots and humans.

Figure 2: A, B, C and D: density plots for time activity features in both
classes; dotted lines indicate the mean for each group. E and F: scatter plots
for time activity features. $n$ and $m1$ are in log scale.

In order to visualize qualitative differences between the behaviors in both
groups (normal and suspect accounts) we show in fig. 2 density and scatter
plots for some of the temporal features. First we select the two temporal
features with higher importance: time entropy (panel A) and number of statuses
(panel B). While modes are visible different in both cases, distributions of
both groups seem to be considerably overlapped. This overlapping is more
evident for the distribution of the first moment (mean of time interval
distribution, panel C) and for the distribution of the coefficient of
variation (defined as $\sqrt{m2-m1^{2}}/m1$). On panel E we show a scatter
plot of n vs time entropy; at first sight, there is no clear distinction
between both groups. The same thing happens with a scatter plot $m1$ and
coefficient of variation; accounts with highest normalized variability are
normal accounts (gray dots), but the two groups overlap all other this phase
space.

For our unsupervised analysis, we chose $k=3$ for a k-nearest neighbors
clustering according to the elbow criterion. When we look at non-temporal
features (density for the fraction of followers and followers vs friends
scatter plot in fig. 3 A and B) we see that all three groups mix together,
indicating a lack of correlation between temporal and non-temporal features.

Figure 3: A: density plot for followers ratio for the three groups we found by
K-mean clustering. B: scatter plot for followers vs friends (the two most
important features according to the random forest classifier.

As an additional analysis, we constructed a retweet graph for the activity of
$\\#$StopTheSteal between November 5 and November 9, 2020. We selected
accounts that tweeted four or more times in this period and established that
two accounts are connected if one was retweeted by the other. This gives us a
directed network where we can visualize structure within these digital
communities. We show in fig. 4 this network, with normal accounts in green and
suspect accounts in red. Again, we see that both groups are completely mixed,
indicating that suspect accounts are present all over the network.

## Conclusions

All results above discussed suggest that political spambots mimic normal,
human temporal-behavior very well, thus putting into the question the validity
of using time features to detect political spambots. This is a novel finding,
since these kind of time activity features used to be useful for detecting
bots on digital social networks. However, these fake accounts are constantly
evolving, showing behaviours that are every time more similar to that of
organic, human accounts. These results indicate the need to develop more
sophisticated algorithms, techniques and tools to automatically detect and
filter out these malicious account that pollute organic conversation on
digital social networks and can potentially manipulate public opinion.

Figure 4: Retweet graph for $\\#$StopTheSteal between November 5 and November
9, 2020. Nodes are Twitter accounts and they are connected if there is a
retweet relationship between them. Green nodes are normal accounts and red
nodes are suspect accounts. White nodes are accounts that are no longer
available for analysis.

## Acknowledgments

OF was a grant holder of the Dirección General de Asuntos del Personal
Académico, Universidad Nacional Autónoma de México - Postdoctoral Scholarships
Program at CEIICH, UNAM.

## References

  * [1] A. Al-Rawi, J. Groshek, and L. Zhang.  What the fake? assessing the extent of networked political spamming and bots in the propagation of# fakenews on twitter.  Online Information Review, 2019.
  * [2] L. Alonso-Muñoz, S. Marcos-García, and A. Casero-Ripollés.  Political leaders in (inter) action. twitter as a strategic communication tool in electoral campaigns.  Trípodos, (39):71–90, 2016.
  * [3] C. Bandt and B. Pompe.  Permutation entropy: a natural complexity measure for time series.  Physical review letters, 88(17):174102, 2002.
  * [4] D. M. Beskow and K. M. Carley.  Bot conversations are different: leveraging network metrics for bot detection in twitter.  In 2018 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM), pages 825–832. IEEE, 2018.
  * [5] N. Chavoshi, H. Hamooni, and A. Mueen.  Identifying correlated bots in twitter.  In International conference on social informatics, pages 14–21. Springer, 2016.
  * [6] N. Chavoshi, H. Hamooni, and A. Mueen.  Temporal patterns in bot activities.  In Proceedings of the 26th international conference on world wide web companion, pages 1601–1606, 2017.
  * [7] Z. Chu, S. Gianvecchio, H. Wang, and S. Jajodia.  Detecting automation of twitter accounts: Are you a human, bot, or cyborg?  IEEE Transactions on dependable and secure computing, 9(6):811–824, 2012.
  * [8] M. Conover, J. Ratkiewicz, M. Francisco, B. Gonçalves, F. Menczer, and A. Flammini.  Political polarization on twitter.  In Proceedings of the International AAAI Conference on Web and Social Media, volume 5, 2011.
  * [9] B. A. Conway, K. Kenski, and D. Wang.  The rise of twitter in the political campaign: Searching for intermedia agenda-setting effects in the presidential primary.  Journal of Computer-Mediated Communication, 20(4):363–380, 2015\.
  * [10] S. Cresci, R. Di Pietro, M. Petrocchi, A. Spognardi, and M. Tesconi.  Fame for sale: Efficient detection of fake twitter followers.  Decision Support Systems, 80:56–71, 2015.
  * [11] L. Dang-Xuan, S. Stieglitz, J. Wladarsch, and C. Neuberger.  An investigation of influentials and the role of sentiment in political communication on twitter during election periods.  Information, communication & society, 16(5):795–825, 2013.
  * [12] C. A. Davis, O. Varol, E. Ferrara, A. Flammini, and F. Menczer.  Botornot: A system to evaluate social bots.  In Proceedings of the 25th international conference companion on world wide web, pages 273–274, 2016.
  * [13] A. Davoudi, A. Z. Klein, A. Sarker, and G. Gonzalez-Hernandez.  Towards automatic bot detection in twitter for health-related tasks.  AMIA Summits on Translational Science Proceedings, 2020:136, 2020\.
  * [14] A. Duh, M. Slak Rupnik, and D. Korošak.  Collective behavior of social bots is encoded in their temporal twitter activity.  Big data, 6(2):113–123, 2018.
  * [15] P. G. Efthimion, S. Payne, and N. Proferes.  Supervised machine learning bot detection techniques to identify social twitter bots.  SMU Data Science Review, 1(2):5, 2018.
  * [16] G. S. Enli and E. Skogerbø.  Personalized campaigns in party-centred politics: Twitter and facebook as arenas for political communication.  Information, communication & society, 16(5):757–774, 2013.
  * [17] E. Ferrara, O. Varol, F. Menczer, and A. Flammini.  Detection of promoted social media campaigns.  In Proceedings of the International AAAI Conference on Web and Social Media, volume 10, 2016.
  * [18] R. Ghosh, T. Surachawala, and K. Lerman.  Entropy-based classification of’retweeting’activity on twitter.  arXiv preprint arXiv:1106.0346, 2011.
  * [19] S. Hegelich and D. Janetzko.  Are social bots on twitter political actors? empirical evidence from a ukrainian social botnet.  In Proceedings of the International AAAI Conference on Web and Social Media, volume 10, 2016.
  * [20] P. Howard, B. Kollanyi, and S. C. Woolley.  Bots and automation over twitter during the second us presidential debate.  2016\.
  * [21] J. Rodríguez-Ruiz, J. I. Mata-Sánchez, R. Monroy, O. Loyola-Gonzalez, and A. López-Cuevas.  A one-class classification approach for bot detection on twitter.  Computers & Security, 91:101715, 2020.
  * [22] C. Shao, G. L. Ciampaglia, O. Varol, K.-C. Yang, A. Flammini, and F. Menczer.  The spread of low-credibility content by social bots.  Nature communications, 9(1):1–9, 2018.
  * [23] C. Shao, P.-M. Hui, L. Wang, X. Jiang, A. Flammini, F. Menczer, and G. L. Ciampaglia.  Anatomy of an online misinformation network.  PloS one, 13(4):e0196087, 2018.
  * [24] S. Stier, A. Bleier, H. Lietz, and M. Strohmaier.  Election campaigning on social media: Politicians, audiences, and the mediation of political communication on facebook and twitter.  Political communication, 35(1):50–74, 2018.
  * [25] K. Vonderschmitt.  The growing use of social media in political campaigns: How to use facebook, twitter and youtube to create an effective social media campaign.  2012\.
  * [26] F. Wei and U. T. Nguyen.  Twitter bot detection using bidirectional long short-term memory neural networks and word embeddings.  In 2019 First IEEE International Conference on Trust, Privacy and Security in Intelligent Systems and Applications (TPS-ISA), pages 101–109. IEEE, 2019.


XX XX X November Special Issue on the Wellbeing for Resilience: Developers
Thrive

Theme Article: Special Issue on the Wellbeing for Resilience: Developers
Thrive

# Toward Optimal Psychological Functioning in AI-driven Software Engineering
Tasks: The SEWELL-CARE Assessment Framework

Oussama Ben Sghaier  Université de Montréal, Canada  Jean-Sébastien Boudrias
Université de Montréal, Canada  Houari Sahraoui  Université de Montréal,
Canada

(2023)

###### Abstract

In the field of software engineering, there has been a shift towards utilizing
various artificial intelligence techniques to address challenges and create
innovative tools. These solutions are aimed at enhancing efficiency,
automating tasks, and providing valuable support to developers. While the
technical aspects are crucial, the well-being and psychology of the
individuals performing these tasks are often overlooked. This paper argues
that a holistic approach is essential, one that considers the technical,
psychological, and social aspects of software engineering tasks. To address
this gap, we introduce SEWELL-CARE, a conceptual framework designed to assess
AI-driven software engineering tasks from multiple perspectives, with the goal
of customizing the tools to improve the efficiency, well-being, and
psychological functioning of developers. By emphasizing both technical and
human dimensions, our framework provides a nuanced evaluation that goes beyond
traditional technical metrics.

††journal: IEEE Software

## Introduction

The rise of artificial intelligence marks a profound transformation in the
role of software developers. A plethora of AI-driven assistants has been
developed and evolved to support developers in their tasks [1]. These
advancements in AI have empowered us to tackle a broader spectrum of software
engineering tasks, address more intricate challenges, and elevate the overall
degree of automation. This has enabled researchers to propose sophisticated
tools endowed with assistive or automation capacities that exhibit heightened
accuracy, intelligence, automation, and efficiency. The goal is to enhance the
efficiency of these tasks and mitigate prevailing challenges.

However, software engineering problems were predominantly addressed from a
technical perspective. The conventional approach for evaluating the proposed
tools, in software engineering, has been mainly centered on technical metrics
[2]. For instance, performance metrics such as precision, recall, BLEU score,
etc. are commonly used as objective measures to assess the performance and
efficacy of AI-based tools.

The prevailing tendency to prioritize technical evaluations leads to
overlooking the subjective experience, psychology, and cognitive needs of the
user. However, the proposed tools are mainly utilized by humans (e.g.,
developers). That is, human aspects constitute an essential dimension that is
often underexplored. This suggests the inadequacy of purely technical
evaluation, as such an assessment remains incomplete, rendering the tools
impractical and ineffectual in real-world scenarios. The pivotal role of the
user accentuates the need to extend the evaluation beyond mere technical
parameters [3]. Consequently, a comprehensive and equitable assessment
necessitates the inclusion of human factors associated with software
engineering tasks, encompassing elements related to the well-being of the
user. This entails considering elements such as user satisfaction, stress
levels, commitment, autonomy, well-being, etc. which are paramount to have a
complete and fair evaluation. Incorporating these factors not only contributes
to the development of more robust and adapted tools but also fosters a
conducive work environment.

Promoting employee well-being has been shown to enhance work engagement [4],
support employee autonomy [5], foster organizational citizenship behaviors
[6], and improve productivity [7]. Thus, the performance and reliability of
the produced software should not be the only criteria that guide the
development of AI-based tools. Yet, aspects of human development for the
different software engineering actors should also be considered The trade-off
between technical aspects and human factors must be carefully navigated to
ensure that the software engineering tools not only enhance efficiency but
also support the well-being and optimal psychological functioning of those
involved.

Optimal psychological functioning is an area of study that focuses on
understanding how individuals become the best that they can be and how to
achieve their full potential [8]. It reflects the paradigm of positive
psychology and is concerned with a person’s achievement of maximization in his
functioning, whether mental, cognitive, emotional, or social [9].

In this paper, we propose a comprehensive and holistic assessment framework
called SEWELL-CARE (_Software Evaluation for WELL-being and optimal
psychological functioning in a Context-AwaRe Environment_). SEWELL-CARE
employs different attributes to accurately characterize AI-driven software
engineering tasks. It also presents the potential outcomes that may have an
impact on developers or other tasks. This enables a fair evaluation of
software engineering tasks that complements technical measures with
psychological and human aspects. Given a specific context, SEWELL-CARE allows
the design of adequate AI-based assistants with the aim of maximizing the
desired outcomes. This makes it possible to build complete tools that
accurately assist developers in their tasks, ensure their well-being, and
allow them to achieve optimal psychological functioning.

We use a real-world scenario to illustrate the usage of SEWELL-CARE. We
investigate the relation between selected subsets of characteristics and
outcomes. Then, we discuss the optimal task design to maximize the desired
outcomes.

The remainder of this paper is organized as follows. First, we present the
proposed assessment framework. Second, we illustrate the usage of our
framework with a real-world scenario. Third, we outline some related work.
Finally, we conclude.

## SEWELL-CARE: An assessment framework for AI-driven software engineering
tasks

We propose a conceptual framework, named SEWELL-CARE, for assessing AI-driven
software engineering tasks. In this framework, we consider the impact of the
AI-driven task and its associated tool on the well-being and psychological
functioning of the developer. This impact is also influenced by factors
related to the developer and the environment. Figure 1 presents a schematic
representation of SEWELL-CARE.

Figure 1: SEWELL-CARE: An assessment framework for AI-driven software engineering tasks Table 1: Description of the characteristics of software engineering tasks  Characteristic |  Description
---|---
Interdependence |  Degree to which a task is dependent on other tasks/other tasks depend on.
Skill variety |  Degree to which the task requires a variety of skills to be accomplished.
Significance |  Degree to which the task influences people or other tasks.
Identity |  Degree to which a task involves a set of activities that identify the task.
Complexity |  Degree a task is complex and difficult to perform (i.e., requires high-level skills and is more mentally demanding and challenging).
Feedback |  Refers to information, opinions, or evaluations provided by others with the aim of offering insight, guidance, or assessment.
Specialization |  Extent to which a task requires specific knowledge and skills.
Error rate |  Frequency and percentage of mistakes to which a task is tolerant.
Autonomy |  Degree of freedom and independence to accomplish a task (work scheduling, decision making, work methods).
Problem solving |  Degree to which a task requires unique solutions and innovative ideas.
Accountability |  Extent to which a person is responsible for the task.
Automation |  Degree to which a task is automated or involves the use of assistant/automation tools.
Explainability |  The extent to which a developer understands the results provided to him by the tools.
Quality |  Quality of the results generated by the tools.
Relevance |  Relevance of the provided results.
Timeliness |  Refers to how often and how quickly is adequate assistance provided.
Table 2: Description of outcomes Outcome |  Description
---|---
Performance–objective |  Objective performance is based on measurable and quantifiable data. It involves concrete and factual criteria that are not influenced by personal opinions or emotions.
Performance–subjective |  Subjective performance is based on personal opinions, perceptions, or judgments. It relies on the person’s personal feelings, beliefs, and interpretations.
Motivation |  Refers to individual energization, influencing his initiation, persistence, and enthusiasm in pursuing and completing the task successfully.
Satisfaction |  Person’s contentment and fulfillment for the work done in a particular task in accordance with expectations and requirements.
Stress |  Refers to the heightened mental or emotional strain experienced by an individual due to perceived pressure, demands, or challenges associated with the task.
Overload |  Refers to a condition where an individual is subjected to an excessive amount of work or information surpassing their capacity and potentially leading to difficulties in managing and completing tasks effectively.
Task conflict |  Refers to the perceived disagreements with other tasks.
Role ambiguity |  Refers to the perceived uncertainty or lack of clarity regarding one’s task duties, responsibilities, or expectations.
Commitment |  Refers to dedication, involvement, and responsibility to carry out and complete the task.
Rejection |  Implicit or explicit disapproval or refusal to accomplish/be assigned to a task.
Table 3: Description of contextual parameters Parameter |  Description
---|---
Experience |  Refers to the experience of the person who is accomplishing the task.
Knowledge |  Refers to the knowledge and expertise of the person who is doing the task.
Job stability |  Refers to the security and predictability of one’s employment within the organization.
Team support |  Refers to the level of help, assistance, collaboration, and resources provided by team members.
Software infrastructure |  Refers to the foundational software resources and technologies available to support the task, including hardware, code base quality, and infrastructure that can affect task execution.
Scope |  Encompasses the comprehensive framework of the task, which includes factors such as project deadlines, client or internal project distinctions, and the overall context within which the task operates.
Team communication |  Involves the quality, frequency, and effectiveness of communication and coordination among team members.

#### Characteristics

The left side of Figure 1 serves as a representation of task and tool
characteristics. These attributes encapsulate features pertaining to software
engineering tasks. Additionally, we include attributes related to AI tools
used as assistants for these tasks. The delineation of a software engineering
task in practical scenarios is based on the calibration and instantiation of
these properties. Table 1 defines these characteristics.

#### Outcomes

The right side delineates the potential outcomes associated with software
engineering tasks. These cover the possible consequences of a task both on
developers and on other tasks. Table 2 outlines these outcomes. The outcomes
are organized into distinct categories. The category denoted as _performance_
includes technical metrics of both qualitative and quantitative nature. The
_general well-being_ category comprises a spectrum of psychological factors
that influence the overall state of mental health, happiness, and contentment
experienced by individuals when completing a task. The _work dynamics_
category encompasses outcomes that impact other tasks or the work in general.
Lastly, the _attitude_ category covers attributes related to the behavior and
demeanor of practitioners with respect to the task under consideration.

#### Contextual parameters

The relationship between the characteristics and outcomes is contingent upon a
specific context. It depends on the environment as well as the person who is
accomplishing the task. Table 3 details these parameters.

The contextual parameters could be split into two major groups. The _human
parameters_ group refers to the set of attributes that characterize the person
doing the task. The _environment parameters_ group includes attributes that
characterize the job environment.

In our proposed framework, we defined a three-dimensional space comprising
characteristics, contextual parameters, and outcomes. Our primary objective is
to investigate the interrelationship between these three dimensions,
specifically focusing on the correlation between characteristics and outcomes
across various contexts. It is important to note that the analysis of such
relationships is inherently intricate due to the vast number of possible
permutations when working within the constraints of a three-dimensional space.

To facilitate a more manageable analysis, we propose to practitioners a
strategy that involves partitioning the investigation into discrete
components, by considering specific use cases. That is, we need to establish
the context by explicitly setting the human and environment parameters. Within
a particular context or setting, we are equipped to conduct a comprehensive
correlation analysis between the characteristics of a given task and its
potential outcomes. This analysis allows us to pinpoint the characteristics
that significantly impact each outcome and, crucially, to assess the direction
(i.e., positive or negative) and strength of these correlations. Consequently,
by gaining a deeper understanding of these relationships, we can strategically
optimize the design of the associated tasks to achieve the desired outcomes
effectively and efficiently.

## Illustrative example

To demonstrate the practical applicability of SEWELL-CARE, we employ a real-
world scenario involving the code review task. Within this scenario, two
Python developers, named _Vinh_ and _Bakhta_ , are employed by the same
company. They work on a platform for financial data analysis. The two
developers are regularly involved in reviewing the code of their fellows.
_Bakhta_ is an experimented developer, with several years of experience
working for other companies. She has been regularly involved in the process of
code review. _Vinh_ is an entry-level Python developer, representing a more
junior role. He hasn’t previous experience in performing code review.

Each case within the study represents a unique contextual setting. While these
cases share most of the environment parameters, they represent two distinct
profiles. We instantiate the contextual parameters for each use case as
detailed in Table 4.

Table 4: Contextual parameters assessment for the case study  | Parameter | Vinh | Bakhta
---|---|---|---
Human | Knowledge | 1 | 5
Experience | 1 | 4
Environment | Job stability | 2 | 4
Team support | 4 | 4
Software infrastructure | 2 | 2
Team communication | 4 | 4

To set the context, we assign a value to each parameter in a _1-5 rating
scale_. Notably, both developers operate within a common environment, leading
to identical values for the corresponding environment parameters except for
_job stability_. For the human parameters, _Bakhta_ is a more knowledgeable
and experienced developer in comparison to _Vinh_.

### Analysis

For each case, we conduct an in-depth investigation into the influence of
varying levels of these task characteristics (low, medium, high) on the
respective outcomes, employing a correlation matrix. Our analysis is guided by
various research contributions on motivation and tolerance to complexity,
e.g., [10, 11].

To streamline the analysis, we focus on a subset of characteristics and
outcomes. Specifically, we chose to examine the characteristics of
_significance_ , _automation_ , _complexity_ , and _problem solving_. In the
case of code review, AI-driven tools can offer different levels of assistance,
ranging from indicating the presence of a potential issue in the code to
suggesting code changes, including the problem localization in the code, and
composing review comments. Each level of assistance determines the levels of
the considered characteristics. For the outcomes, we select _performance-
objective_ , _stress_ , _motivation_ , and _rejection_.

We investigate the relationship between the characteristics and outcomes.
Table 5 reports the correlations for each case. These correlations offer
valuable insights into how the characteristics differently impact various
outcomes of software engineering tasks according to the given context.

Table 5: Correlation between a subset of characteristics and outcomes

|  Performance-Objective |  Stress |  Motivation |  Rejection
---|---|---|---|---
Bakhta
Significance |  A |  A |  A |  A
Automation |  A |  A |  A |  A
Complexity |  A |  A |  A |  A
Problem solving |  A |  A |  A |  A
Vinh
Significance |  A |  A |  A |  A
Automation |  A |  A |  A |  A
Complexity |  A |  A |  A |  A
Problem solving |  A |  A |  A |  A

A | strongly negatively correlated | A | negatively correlated | A | unrelated | A | correlated | A | strongly correlated
---|---|---|---|---|---|---|---|---|---

#### Significance

Task significance carries varying weight for experienced and novice
developers. For the experienced developer, _Bakhta_ , a task’s significance
resonates as a hallmark of importance, bolstering her performance and
motivation. The greater the significance, the more it piques her interest,
alleviates stress, and fosters a willingness to accept the task.

Conversely, for the novice developer, _Vinh_ , the dynamics shift. The weight
of task significance exerts a different influence. In his case, increased task
significance may, paradoxically, hinder his performance and motivation. As a
less experienced developer, elevated significance can increase stress levels,
ultimately leading to task rejection in cases of heightened importance and
criticality.

#### Automation

In the context of automation, the impact on _Bakhta_ ’s performance is
notable: the higher the level of task automation, the more proficient she
becomes. However, this increase in performance is accompanied by reduced
motivation, as heightened automation tends to render tasks less challenging
and more repetitive, thereby diminishing their intrinsic appeal.
Paradoxically, a substantial increase in task automation may also lead to
elevated stress for _Bakhta_ , as it diminishes her fine-grained control over
the task.

Conversely, for _Vinh_ , the consequences of automation take a different turn.
Elevated task automation enhances both his performance and motivation,
offering valuable assistance and simplifying tasks. This reduction in
complexity and the assistance provided by automation serve to alleviate
stress, making him more inclined to engage in and accept this kind of task
facilitated by automation tools, as it aligns well with his role as a junior
developer.

#### Complexity

_Bakhta_ demonstrates consistent performance across tasks of varying
complexity, handling both easy and complex tasks with equal proficiency.
However, a distinctive pattern emerges in her emotional response: for more
challenging tasks, she experiences increased stress yet heightened motivation.
Her extensive experience in the field renders complex tasks more attractive,
reflecting her keen interest in tackling them.

In contrast, _Vinh_ lacks the extensive experience of his counterpart. Thus,
complex tasks tend to diminish his motivation, heighten his stress levels, and
lead to decreased performance. Consequently, he tends towards rejecting
complex tasks, given that they present greater challenges that could surpass
his current skill set and experience level.

#### Problem solving

_Bakhta_ prefers tasks that demand substantial problem-solving and creative
prowess. Engaging in such tasks may induce a mild level of stress in her, but
it significantly heightens her motivation to successfully complete them.

_Vinh_ tends to avoid assignments that involve high levels of problem solving
and creativity, primarily due to his comparatively lower experience and
skills. These tasks tend to evoke a high degree of stress in him and are
accompanied by reduced motivation.

## Related work

Many works attempted to address work design and work psychology in software
engineering.

In [12], the authors introduce a comprehensive measure known as the Work
Design Questionnaire (WDQ). This WDQ was devised to define and characterize a
job and to measure these characteristics. The identified work characteristics
were categorized into three groups: motivational, social, and contextual
characteristics. Each of these work characteristics was associated with a
distinct set of items, to define and facilitate their measurement through a
straightforward 5-point scaling system. The authors extend this work in [13],
delving into the impact of work characteristics on outcomes. This exploration
was undertaken through a series of hypothesized relationships, that were
validated with a user study.

In [14], the authors introduce _SPACE_ , a multi-dimensional framework for
developer productivity. It captures different dimensions of productivity
including satisfaction, performance, activity, communication, and efficiency.
This enables an understanding of developer productivity and thus ensures the
efficient development of software systems and the well-being of developers.

In [15], the authors propose the _DX Framework_ , an actionable conceptual
framework for understanding and improving developer experience to enable more
productive and effective work environments. The framework elucidates factors
that affect developer experience and characteristics that influence their
respective importance to individual developers. It also outlines barriers that
hinder the developer experience and presents strategies employed by
individuals and teams to improve the developer experience. This conceptual
framework is built to understand and guide developer experience improvements.

While these contributions present practical and insightful perspectives, a
notable void remains unaddressed. None of the extant endeavors has delved into
software engineering tasks, as these works predominantly operate at the
macroscopic job level. Moreover, some of these contributions selectively
focalize their attention on specific outcomes. The focus of these works
primarily centers on developers and the job in a broader sense, without due
consideration for the integral role of AI tools in the execution of these
tasks.

## Conclusion

In this paper, we introduce SEWELL-CARE, a novel conceptual framework tailored
for the evaluation of AI-driven software engineering tasks. This framework
serves to characterize AI-driven software engineering tasks within diverse
contextual settings and to comprehensively encapsulate their potential
outcomes, encompassing psychological dimensions. This comprehensive framework
enables practitioners to strategically tailor task designs, to achieve the
desired outcomes. The overarching aim is to foster the design of well-balanced
tasks, optimizing their efficiency and concurrently ensuring the well-being of
the developers involved in their execution.

The proposed framework currently adopts a descriptive nature, elucidating the
relationships between task characteristics and their associated outcomes. As
part of future work, we aspire to transition from a descriptive stance to a
predictive one. To this end, we plan to concretize these relationships by
developing predictive models that quantitatively gauge outcomes based on the
task characterizations within specific contextual settings. This transition
will empower us to forecast and anticipate outcomes, thereby enhancing the
framework’s utility and its capacity to practically guide optimized task
designs.

## References

  * [1] Y. Yang, X. Xia, D. Lo, and J. Grundy, “A survey on deep learning for software engineering,” _ACM Computing Surveys (CSUR)_ , vol. 54, no. 10s, pp. 1–73, 2022.
  * [2] A. B. Sai, A. K. Mohankumar, and M. M. Khapra, “A survey of evaluation metrics used for nlg systems,” _ACM Computing Surveys (CSUR)_ , vol. 55, no. 2, pp. 1–39, 2022.
  * [3] O. Hazzan and J. Tomayko, “Human aspects of software engineering: The case of extreme programming,” in _International Conference on Extreme Programming and Agile Processes in software Engineering_. Springer, 2004, pp. 303–311.
  * [4] R. Cropanzano and T. A. Wright, “When a" happy" worker is really a" productive" worker: A review and further refinement of the happy-productive worker thesis.” _Consulting Psychology Journal: Practice and Research_ , vol. 53, no. 3, p. 182, 2001.
  * [5] B. M. Staw, R. I. Sutton, and L. H. Pelled, “Employee positive emotion and favorable outcomes at the workplace,” _Organization science_ , vol. 5, no. 1, pp. 51–71, 1994.
  * [6] K. Lee and N. J. Allen, “Organizational citizenship behavior and workplace deviance: the role of affect and cognitions.” _Journal of applied psychology_ , vol. 87, no. 1, p. 131, 2002.
  * [7] M. Patterson, P. Warr, and M. West, “Organizational climate and company productivity: The role of employee affect and employee level,” _Journal of occupational and organizational psychology_ , vol. 77, no. 2, pp. 193–216, 2004.
  * [8] L. A. Chénard-Poirier, R. J. Vallerand, J. Verner-Filion, N. Houlfort, J. Forest, and N. Rinfret, “Optimal functioning in society: A conceptualization, a measure, and a look at determinants,” _Journal of Happiness Studies_ , vol. 24, no. 2, pp. 857–892, 2023.
  * [9] H. P. Phan, H.-W. Wang, J.-H. Shih, S.-Y. Shi, R.-Y. Lin, and B. H. Ngu, “The importance of mindfulness in the achievement of optimal functioning: Conceptualization for research development,” in _Educational psychology-between certitudes and uncertainties_. IntechOpen, 2018.
  * [10] V. S. Ramachandran and W. Hirstein, “The science of art: A neurological theory of aesthetic experience,” _Journal of consciousness Studies_ , vol. 6, no. 6-7, pp. 15–51, 1999.
  * [11] M. Csikszentmihalhi, _Finding flow: The psychology of engagement with everyday life_. Hachette UK, 2020.
  * [12] F. P. Morgeson and S. E. Humphrey, “The work design questionnaire (wdq): developing and validating a comprehensive measure for assessing job design and the nature of work.” _Journal of applied psychology_ , vol. 91, no. 6, p. 1321, 2006.
  * [13] S. E. Humphrey, J. D. Nahrgang, and F. P. Morgeson, “Integrating motivational, social, and contextual work design features: a meta-analytic summary and theoretical extension of the work design literature.” _Journal of applied psychology_ , vol. 92, no. 5, p. 1332, 2007.
  * [14] N. Forsgren, M.-A. Storey, C. Maddila, T. Zimmermann, B. Houck, and J. Butler, “The space of developer productivity: There’s more to it than you think.” _Queue_ , vol. 19, no. 1, pp. 20–48, 2021.
  * [15] M. Greiler, M.-A. Storey, and A. Noda, “An actionable framework for understanding and improving developer experience,” _IEEE Transactions on Software Engineering_ , 2022.

Oussama Ben Sghaier is a Ph.D. student at the Department of Computer Science
and Operations Research of the Université de Montréal at Montréal, Québec,
Canada. His research interests include artificial intelligence for software
engineering, natural language processing, and model-driven engineering.
Contact him at<EMAIL_ADDRESS>
Jean-Sébastien Boudrias is a professor at the Department of Psychology of the
Université de Montréal at Montréal, Québec, Canada. His research interests
include employee empowerment, personality, and psychological health at work.
Contact him at<EMAIL_ADDRESS>
Houari Sahraoui is a professor at the Department of Computer Science and
Operations Research of the Université de Montréal at Montréal, Québec, Canada.
His research interests include artificial intelligence techniques applied to
software engineering, search-based software engineering, and model-driven
engineering. Contact him at<EMAIL_ADDRESS>