# On HRTF Notch Frequency Prediction
Using Anthropometric Features and Neural Networks

###### Abstract

High fidelity spatial audio often performs better when produced using a
personalized head-related transfer function (HRTF). However, the direct
acquisition of HRTFs is cumbersome and requires specialized equipment. Thus,
many personalization methods estimate HRTF features from easily obtained
anthropometric features of the pinna, head, and torso. The first HRTF notch
frequency (N1) is known to be a dominant feature in elevation localization,
and thus a useful feature for HRTF personalization. This paper describes the
prediction of N1 frequency from pinna anthropometry using a neural model.
Prediction is performed separately on three databases, both simulated and
measured, and then by domain mixing in-between the databases. The model
successfully predicts N1 frequency for individual databases and by domain
mixing between some databases. Prediction errors are better or comparable to
those previously reported, showing significant improvement when acquired over
a large database and with a larger output range.

Index Terms—  Head-related transfer function (HRTF), spatial audio, machine
learning, anthropometry, notch,

##  1 Introduction

Binaural reproduction of spatial audio often achieves best results when
personalized using the listener’s unique head-related transfer function (HRTF)
[1]. However, HRTF acquisition is cumbersome and requires specialized
equipment [2]. A need exists for HRTF personalization methods which may be
employed by end-users, not requiring specialized equipment. HRTF is a product
of the pinna, head, and torso physiology [3]. Thus, various approaches for
HRTF personalization are based on prediction from anthropometric features. One
approach involves direct prediction of the entire HRTF, or of distinct aspects
such as magnitude and specific frequency bins [4], which are explicitly used
for reproduction. Another approach is to predict different HRTF features in
order to inform HRTF selection. In this approach highly indicative HRTF
features are predicted. Then, a best matching HRTF is selected from a
database. Across all approaches, prediction is performed using different
anthropometric feature representations, such as raw geometric features [3],
autoencoded features [5], images [6], and 3D representations [7, 8].

Distinct HRTF features seem to correspond to different localization tasks,
thus HRTF feature prediction may be conducted towards specific performance
goals. Horizontal localization is dominated by binaural cues, namely
interaural time difference (ITD) and interaural level difference (ILD). These
cues primarily stem from head shape and so may be straightforwardly
approximated [9]. On the other hand, vertical localization is dominated by
monaural spectral cues originating from pinna features which are reflected in
the HRTF’s high frequencies [10].

Most HRTFs comprise one or more notches in the frequency range above $5\,$kHz,
termed N1, N2, et cetera. The notches’ central frequencies are dominant
vertical localization cues [11]. Therefore, HRTF personalization methods
focusing on vertical localization may rely on notch frequency [12, 13].

Several studies focus on notch frequency prediction from anthropometric
features [12, 14, 15]. Typically, prediction is conducted using small measured
HRTF databases of 20-80 ears and geometric pinna input features. Sometimes,
more elaborate 3D input features are used [13]. However, obtaining these
features likely requires specialized equipment, making them unsuitable for an
end-user. Reported prediction errors are typically in the $0.05-0.1\,$octave
range, while the just noticeable difference (JND) for N1 notch frequency is
$0.1-0.2\,$octave [11]. The small database size affects the prediction
accuracy and limits the prediction method to multivariate linear regression.
Indirectly, small databases are also likely to have a smaller output range,
which in turn somewhat facilitates the prediction, reduces the estimation
error, but diminishes generalizability. Existing literature does not describe
N1 frequency prediction using very large databases with a sufficiently large
output range, or by machine learning methods from simple pinna features.

This paper presents N1 frequency prediction from pinna geometry using three
HRTF databases: CHEDAR - a large parameterized-simulated database [16], HUTUBS
- a small database of both simulated and measured data [17], and a small,
measured, internal database. The databases are larger, and comprise a larger
output range, than those used in existing works. N1 frequency is predicted by
a neural model for each database separately. In addition, domain mixing is
employed to enhance predictions on the small databases, with the goal of
enabling future listening tests. The neural model outperforms linear
regression in most cases when tested over a large number of naive subjects
spanning a large output range. Domain mixing further improves predictions for
both HUTUBS and the internal database. The results indicate that N1 frequency
may be predicted in high accuracy given sufficiently large databases, and that
prediction may be performed on small databases by domain mixing, under
specific conditions.

##  2 Data Preparation

Model | CHEDAR |  | HUTUBS
---
(simulated)
| HUTUBS
---
(measured)
M1
| RMS Hz | octave | RMS Hz | octave | RMS Hz | octave | RMS Hz | octave
Naive prediction | 1151 | 0.152 | 1014.19 | 0.13 | 1075.1 | 0.16 | 927.2 | 0.15
Linear | 953.3 | 0.17 | 1060.68 | 0.15 | 1045.2 | 0.17 | 922.19 | 0.15
Neural | 505.07 | 0.05 | 1033.64 | 0.15 | 1047.86 | 0.17 | 969.82 | 0.14
Domain mixing (CHEDAR) | N/A | N/A | 980.5 | 0.15 | 1105.23 | 0.175 | 1175.69 | 0.19
Domain mixing (HUTUBS) | N/A | N/A | N/A | N/A | N/A | N/A | 896.99 | 0.14
Table 1: N1 frequency prediction errors across databases and models. In some
cases, the neural model outperforms linear prediction. Domain mixing enhances
predictions when both source and target domain are acquired by the same
method.

###  2.1 Databases

HRTF prediction problems pose an inherent trade-off. Databases acquired from
human subjects are useful for conducting listening tests. However, such
databases are typically small, and limit the complexity of possible prediction
models. On the other hand, large databases are typically parameterized, and
thus unsuitable for listening tests. In an attempt to mitigate this trade-off,
this work involves three HRTF databases: CHEDAR, HUTUBS, and an internal
database, denoted as “M1”. Prediction is performed for each database
separately, and domain mixing is performed in-between the databases.

CHEDAR is a simulated parameterized database of 1253 subjects with
corresponding anthropometric data. The head and torso are identical across all
subjects, and the right and left ears are identical for each subject. The
anthropometric data consists of standard CIPIC features: seven pinna distances
($d_{1}-d_{7}$) and two pinna angles (rotation and flare) [18].

HUTUBS is a database acquired from human subjects, consisting of simulated and
measured HRIRs for both ears, and standard CIPIC anthropometric features.
HUTUBS’s total size is 182 ears, excluding repetitions, dummy heads and
subjects without corresponding anthropometric data. M1 is an internal database
consisting of measured HRIRs of both ears of 96 human subjects. In addition,
M1 contains anthropometric data in the form of 53 keypoint coordinates per
ear, extracted from a 3D scan of the subject’s pinnas, and the two standard
CIPIC pinna angles.

For both HUTUBS and M1, both left and right ears are utilized for prediction,
maximizing the databases’ size. Note that mixing right and left ears is made
possible due to the fact that head and torso are known not to affect notch
frequency [19], and that the head and torso’s effect on the HRIR is removed
during the notch extraction process.

###  2.2 Input Features

A total of 9 CIPIC standard input features are used: seven pinna distances
($d_{1}-d_{7}$), and two pinna angles (rotation and flare). While for CHEDAR
and HUTUBS all features are explicitly provided, for M1 the distances are
calculated from a set of keypoint coordinates on the pinna. However, since the
keypoints were not originally positioned with CIPIC features in consideration,
they only approximate CIPIC distances, as described in detail in Section 5.

###  2.3 Output

N1 frequency was extracted from all databases using the method described in
[11], for the front direction HRIR (azimuth $0\degree$, elevation $0\degree$)
of all unique ears. The extraction method consists of clipping the HRIR by a
window centered around the HRIR’s maximum, and performing FFT on the result.
The resulting spectrum’s maximum is recorded as P1, and the lowest local
minimum above P1 is recorded is N1.

Only notch frequencies above $5\,$kHz are known to contribute to elevation
localization. In addition, some subjects lack an apparent notch. Therefore,
subjects with notches lower than $5\,$kHz or without a prominent notch were
removed from the datasets, resulting in a total number of examples of 903
(CHEDAR), 182 (HUTUBS simulated), 171 (HUTUBS measured), and 191 (M1). The N1
frequency in each datasets ranges from $6\,$kHz to over $11\,$kHz ($\sim
0.9\,$octave). For HUTUBS, a discrepancy exists between N1 frequency values
extracted from measured and simulated HRIRs of each subject. The average RMS
difference is $1560\,$Hz, and the correlation coefficient is $0.32$.

##  3 Methods

###  3.1 Single Database Prediction

The N1 frequency was separately predicted for each database using both linear
regression and a neural network. Linear regression was performed to compare
against existing results in literature, and to establish a benchmark for
neural model prediction. Neural model prediction was performed by a model
consisting of three fully connected hidden layers of 40 units each (CHEDAR,
M1) or 20 units each (HUTUBS), shown in Figure 1. The model was trained using
60% of the examples, with 20% used as a validation set, and the remaining
examples used as a test set.

Fig. 1: The neural model used for N1 prediction, consisting of three fully
connected hidden layers.

In order to asses how database size effects prediction performance, the neural
model prediction was also performed with increasing training set sizes, and a
fixed-size test set of 25 examples. For CHEDAR, the training set size ranged
from $50-700$ examples. For HUTUBS and M1, the training set ranged from
$50-150$ examples.

###  3.2 Domain Mixing

Both HUTUBS and M1 are insufficiently large to obtain predictions below JND,
as described in Section 4. This limitation may be mitigated by domain mixing
[20]. Several domain mixing configurations were performed: CHEDAR as source
domain and HUTUBS as target domain; CHEDAR as source domain and M1 as target
domain; and HUTUBS (measured) as source domain and M1 as target domain. For
each configuration, the target domain training set was duplicated, with each
example appearing twice, and mixed with the source domain training set. The
model was then trained on the mixed training set, and tested on the target
domain test set.

##  4 Results

Errors of N1 frequency prediction for all databases and prediction methods are
shown in Table 1. The values represent test set averages over nine executions
with varied randomization. The results are also compared to naive prediction -
prediction of the database’s mean output as a fixed value. For CHEDAR, linear
regression resulted in $953.3\,$Hz RMS error ($0.17\,$octave) and neural model
prediction in $505.07\,$Hz RMS error ($0.05\,$octave). The neural model’s
errors are significantly lower than the lower JND threshold of $0.1\,$ octave.
For CHEDAR, the neural model’s predictions are comparable or better than
predictions reported in literature [11, 12], derived from simple pinna
features, and pertain to a much larger test set.

For HUTUBS, comparable results are obtained over the simulated and measured
HRIRs. For HUTUBS (simulated), the neural model marginally outperforms linear
regression by approximately $30\,$Hz RMS , but does not outperform naive
prediction. For HUTUBS (measured), linear regression and neural prediction
achieve similar results, about $30\,$Hz RMS lower than naive prediction. For
M1, linear regression outperforms both neural prediction and naive prediction.
For both HUTUBS and M1, the overall errors are higher than the lower JND
threshold.

Prediction errors versus training set size are shown in Figure 2. Prediction
errors decrease as the training set size increases, across all databases. For
CHEDAR, 200-300 examples are required to meet the lower JND threshold, as
discussed further in Section 5. This size exceeds the entire small databases.
Figure 2 also includes additional data points for M1 and HUTUBS, which
represent a $180$-example training set achieved by a leave-one-out approach.
These points are not directly comparable to the rest of the data points.
However, they demonstrate the potential of further increasing the training set
size.

Fig. 2: N1 prediction errors versus training set size. The errors decrease
with size. For CHEDAR, 200-300 examples are required to meet lower JND
threshold, while M1 and HUTUBS are insufficiently large.

Domain mixing yielded varied results. With CHEDAR as source domain and HUTUBS
(simulated) as target domain, it outperforms the neural model by approximately
$50\,$Hz. However, for HUTUBS (measured) as target domain, the performance is
worse than the neural model’s, and for M1 it is the worst of all methods.
However, domain mixing with HUTUBS (measured) as source domain and M1 as
target domain outperforms all other methods.

##  5 Discussion

The results obtained with CHEDAR confirm the relation between N1 frequency and
pinna features, and demonstrate that this relation is non-linear.
Theoretically, given mean prediction errors of these magnitudes and a database
acquired on human subjects, HRTF selection and listening tests may be
performed. However, accuracy with HUTUBS and M1 is significantly lower and
insufficient for HRTF selection.

The performance differences may be explained by several reasons. Primarily,
database size seems to be crucial. About 200-300 examples are required for
sufficiently accurate N1 frequency prediction, while most measured databases
are smaller. In addition, simulated databases may be easier to predict than
measured databases in some cases. HRIR measurements and corresponding N1
frequency values may have inaccuracies stemming from the acquisition process.
It is also possible that some simulations over-simplify the relation between
pinna features and HRIR, and are therefore easier to predict. The differences
between simulations and measurements are the only explanation for the
performance distinction observed between HUTUBS’s measured and simulated HRIRs
for some training set sizes.

An additional reason may explain M1’s high errors: M1’s keypoint positions do
not fully correspond to CIPIC distances, and are thus only approximations of
CIPIC features. M1’s features might be comparatively less pertinent to notch
prediction than CIPIC features.

Fig. 3: Feature $d_{3}$ (cavum concha width) distributions for different
databases. HUTUBS’s distribution is partly contained within CHEDAR’s, while
M1’s range is considerably higher. This and similar discrepancies may explain
the inconsistent domain mixing results.

Domain mixing with CHEDAR as source domain is successful on HUTUBS
(simulated), but not on M1. This may be attributed to domain shift. For all
features, HUTUBS distributions are mostly contained within CHEDAR
distributions. However, M1’s distributions are significantly different than
CHEDAR’s for some features. For features $d_{2},d_{3}$ and $d_{7}$, there is
little to no overlap of distributions, as demonstrated in Figure 3. For
features $d_{4},d_{6}$, rotation and flare, the distributions partly overlap.
Only for features $d_{1}$ and $d_{5}$, the distributions considerably overlap.
The domain shift may stem from the incompatible placement of keypoints in M1
as discussed above, or simply reflect the databases’ population variance.

The output acquisition method (simulations or measurements) also seems to be a
key factor. When both the source and target domain were acquired by the same
method, domain mixing outperforms all other methods. However, when the source
and target domain are of different acquisition methods, domain mixing does not
offer an advantage over other methods.

These factors may be considered when preparing databases for HRTF predictions.
Feature distributions of simulated databases may be selected in the first
place to overlap with existing measured databases. Measured databases may be
created with keypoints corresponding to standard pinna features, primarily
CIPIC. It is possible that domain mixing conducted on databases meeting these
requirements would achieve better results. For the purpose of listening tests
on M1, other databases for domain mixing may be explored. Such databases may
be simulated and with a better feature compatibility, or measured with a
larger number of examples.

##  6 Conclusion

This paper presented a neural model for N1 frequency prediction from pinna
features. The model was employed on three databases, consisting of both
measured and simulated HRIRs. Results demonstrate that in certain
configurations, notch frequency may be predicted to be below the JND given a
sufficiently large database, estimated at about $200-300$ examples. Prediction
errors on small databases were further reduced by domain mixing with examples
of other database acquired by the same method. However, domain mixing in-
between simulated and measured databases did not improve the predictions. As
the model uses raw pinna distances and angles, it may be potentially employed
by end users of spatial audio products to inform HRTF personalization. In this
scenario, the features will be extracted from an ear photo acquired by the
user. Further work should focus on successful domain mixing between measured
databases, either by obtaining larger databases or by mixing additional
databases. Given successful predictions on measured databases, subjective
listening tests should then be performed in order to validate N1 frequency as
a basis for HRTF personalization. In addition, the prediction may be expanded
to include other elevations and azimuths.

## References

  * [1] Corentin Guezenoc and Renaud Seguier,  “HRTF individualization: A survey,”  in Audio Engineering Society Convention 145. Audio Engineering Society, 2018.

  * [2] Areti Andreopoulou, Durand R. Begault, and Brian F.G. Katz,  “Inter-laboratory round robin HRTF measurement comparison,”  IEEE Journal of selected topics in signal processing, vol. 9, no. 5, pp. 895–906, 2015.
  * [3] Robert Pelzer, Manoj Dinakaran, Fabian Brinkmann, Steffen Lepa, Peter Grosche, and Stefan Weinzierl,  “Head-related transfer function recommendation based on perceptual similarities and anthropometric features,”  The Journal of the Acoustical Society of America, vol. 148, no. 6, pp. 3809–3817, 2020.
  * [4] Piotr Bilinski, Jens Ahrens, Mark R.P. Thomas, Ivan J. Tashev, and John C. Platt,  “HRTF magnitude synthesis via sparse representation of anthropometric features,”  in 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2014, pp. 4468–4472.
  * [5] Tzu-Yu Chen, Tzu-Hsuan Kuo, and Tai-Shih Chi,  “Autoencoding HRTFs for DNN based HRTF personalization using anthropometric features,”  in 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2019, pp. 271–275.
  * [6] Bowen Zhi, Dmitry N. Zotkin, and Ramani Duraiswami,  “Towards fast and convenient end-to-end HRTF personalization,”  in 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022, pp. 441–445.
  * [7] Michaela Warnecke, Sharon Jamison, Sebastian Prepelita, Paul Calamia, and Vamsi Krishna Ithapu,  “HRTF personalization based on ear morphology,”  in Audio Engineering Society Conference: AES 2022 International Audio for Virtual and Augmented Reality Conference. Audio Engineering Society, 2022.
  * [8] Yaxuan Zhou, Hao Jiang, and Vamsi Krishna Ithapu,  “On the predictability of HRTFs from ear shapes using deep networks,”  in 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2021, pp. 441–445.
  * [9] Bosun Xie,  Head-related transfer function and virtual auditory display,  J. Ross Publishing, 2013.
  * [10] Kazuhiro Iida, Motokuni Itoh, Atsue Itagaki, and Masayuki Morimoto,  “Median plane localization using a parametric model of the head-related transfer function based on spectral cues,”  Applied Acoustics, vol. 68, no. 8, pp. 835–850, 2007.
  * [11] Kazuhiro Iida, Yohji Ishii, and Shinsuke Nishioka,  “Personalization of head-related transfer functions in the median plane based on the anthropometry of the listener’s pinnae,”  The Journal of the Acoustical Society of America, vol. 136, no. 1, pp. 317–333, 2014.
  * [12] Marius George Onofrei, Riccardo Miccini, Runar Unnthorsson, Stefania Serafin, and Simone Spagnol,  “3D ear shape as an estimator of HRTF notch frequency,”  in 17th Sound and Music Computing Conference. Axea sas/SMC Network, 2020, pp. 131–137.
  * [13] Simone Spagnol, Riccardo Miccini, Marius George Onofrei, Runar Unnthorsson, and Stefania Serafin,  “Estimation of spectral notches from pinna meshes: Insights from a simple computational model,”  IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 29, pp. 2683–2695, 2021.
  * [14] Kazuhiro Iida, Orie Nishiyama, and Tsubasa Aizaki,  “Estimation of the category of notch frequency bins of the individual head-related transfer functions using the anthropometry of the listener’s pinnae,”  Applied Acoustics, vol. 177, pp. 107929, 2021.
  * [15] Riccardo Miccini and Simone Spagnol,  “Estimation of pinna notch frequency from anthropometry: An improved linear model based on principal component analysis and feature selection,”  in Nordic Sound and Music Computing Conference 2019 and the Interactive Sonification Workshop 2019. KTH Royal Institute of Technology, 2019, pp. 5–8.
  * [16] Slim Ghorbal, Xavier Bonjour, and Renaud Seguier,  “Computed HRIRs and ears database for acoustic research,”  in Audio Engineering Society Convention 148. Audio Engineering Society, 2020.
  * [17] Fabian Brinkmann, Manoj Dinakaran, Robert Pelzer, Joschka J. Wohlgemuth, Fabian Seipl, and Stefan Weinzierl,  “The HUTUBS HRTF database,” 2019.
  * [18] Ralph V. Algazi, Richard O. Duda, Dennis M. Thompson, and Carlos Avendano,  “The CIPIC HRTF database,”  in Proceedings of the 2001 IEEE Workshop on the Applications of Signal Processing to Audio and Acoustics (Cat. No. 01TH8575). IEEE, 2001, pp. 99–102.
  * [19] Kazuhiro Iida and Masato Oota,  “Median plane sound localization using early head-related impulse response,”  Applied Acoustics, vol. 139, pp. 14–23, 2018.
  * [20] Karl Weiss, Taghi M. Khoshgoftaar, and DingDing Wang,  “A survey of transfer learning,”  Journal of Big Data, vol. 3, no. 1, pp. 1–40, 2016.


11affiliationtext: Department of Mathematics, Ecole Polytechnique Fédérale de
Lausanne (EPFL), Switzerland22affiliationtext: Centre for Evidence-Based
Medicine Odense (CEBMO) and Cochrane Denmark, Department of Clinical Research,
University of Southern Denmark, Odense, Denmark

# Mindel C. Sheps: Counted, Dead or Alive

Anders Huitfeldt

(August 27, 2024)

##  1 Introduction

When deciding whether to recommend that their patients initiate a medical
treatment (such as the administration of drug $A$ which may affect outcome
$Y$), many clinicians prefer to think about the expected consequences in terms
of the relative risk, that is, a multiplicative parameter that relates the
risk that the patients would experience the outcome if treated (which we will
denote $Pr[Y^{a=1}=1]$), to the risk that they would experience the outcome if
untreated ($Pr[Y^{a=0}=1]$). For example, the Cochrane Handbook(Deeks et al.,
2020) and the GRADE guidelines(Guyatt et al., 2013) both explicitly instruct
clinicians to combine the patient-specific baseline risk with the relative
risk in order to produce an estimate of a patient’s risk under treatment.

While many epidemiologists take a perspective on effect measures that is
consistent with the clinicians’ view, statisticians have tended to be uneasy
about data analyses that rely on relative risk models, in part because such
analyses may lead to predictions outside the range of valid probabilities, and
in part because the models may lead to different predictions depending on
whether they are based on the probability of the outcome event (say, death),
or the probability of the complement of the outcome event (say, being
alive)(Cox, 1972). Stated more formally, the relative risk model is not closed
on the interval [0,1] and not invariant to choice of reference level for the
outcome. These limitations of relative risk models have contributed to a
longstanding difference in opinion between thought leaders from the three
academic traditions, which frequently re-erupts in the scientific
literature(Sackett et al., 1996; Walter, 1998), on social media and on
academic discussions forums.

A simple solution to these limitations has been known since the 1950s but is
not widely known or utilized. This solution was first described by Canadian
physician and biostatistician Mindel Cherniack Sheps (1913-1973) in Shall we
count the living or the dead? (Sheps, 1958), a landmark paper that was
published in the New England Journal of Medicine, and expanded upon with
additional theoretical detail by the same author in Biometrics(Sheps, 1959).
Remarkably, Sheps’ approach not only resolves the theoretical problems of
closure and invariance, it also grounds choice of effect measure in biological
mechanisms consistent with influential work from toxicology, psychology and
philosophy. Variations of Sheps’ insight have been repeatedly rediscovered
over the last 65 years (Weinberg, 1986; Cheng, 1997; Bouckaert and Mouchart,
2001; Huitfeldt et al., 2018; Baker and Jackson, 2018), but remain largely
unknown among clinicians, epidemiologists and other practitioners of applied
statistics.

In this commentary, we refer to the variant of the relative risk that is based
on the probability of the outcome event as the standard risk ratio
($RR=\frac{Pr[Y^{a=1}=1]}{Pr[Y^{a=0}=1]}$) and to the variant that is based on
the complement of the outcome event as the survival ratio
($SR=\frac{Pr[Y^{a=1}=0]}{Pr[Y^{a=0}=0]}$).

##  2 Sheps’ argument

Sheps’ premise is stated very clearly in the introduction to her 1958 paper:
“When we wish to estimate the beneficial effect of a therapeutic agent, or the
harmful effect of a factor such as exposure to radiation, we have the choice
of a number of methods. If a new treatment is associated with lower mortality
rates, we can estimate the beneficial effect from the absolute increase in the
number of survivors, from the percentage increase in the living, or from the
percentage decrease in the dead. Often, these different methods provide
estimates of quite different magnitude. (…) However, in the assessment of
beneficial or harmful effects respectively, there may be cogent reasons for
choosing particular methods”.

To illustrate the setting of beneficial effects, she considers the Polio
vaccine, and shows that, under an assumption of monotonicity (i.e. that the
vaccine does not cause Polio in any individual), the relative decrease in
probability of polio, which with counterfactual notation would be written as
$\frac{Pr[Y^{a=0}=1]-Pr[Y^{a=1}=1]}{Pr[Y^{a=0}=1]}$, can be interpreted as
reflecting the proportion in whom the vaccine prevents polio (i.e. the
prevalence of the “preventive” response type, as described in Greenland and
Robins (1986)) among those who would get polio if unvaccinated. In contrast,
the relative increase in probability of not getting polio
($\frac{Pr[Y^{a=1}=0]-Pr[Y^{a=0}=0]}{Pr[Y^{a=0}=0]}$) has no interpretation in
terms of prevalence of response types. The relative decrease in probability of
the outcome event is equal to $1-RR$, modelling the relative decrease is
therefore equivalent to modelling the standard risk ratio.

Similarly, to illustrate the setting of harmful effects, she considers the
effect of smoking on mortality, and shows that under a monotonicity
assumption, the relative decrease in probability of survival
($\frac{Pr[Y^{a=0}=0]-Pr[Y^{a=1}=0]}{Pr[Y^{a=0}=0]}$) can be interpreted as
the proportion who are harmed (i.e. the prevalence of the “causal” response
type) among those who would have survived if they didn’t smoke. In contrast,
the relative increase in probability of death
($\frac{Pr[Y^{a=1}=1]-Pr[Y^{a=0}=1]}{Pr[Y^{a=0}=1]}$) has no interpretation in
terms of prevalence of response types. The relative decrease in the
probability of the complement of the outcome event is equal to 1-SR, modelling
the relative increase is therefore equivalent to modelling the survival ratio.

Analogous arguments will hold in many settings: Beneficial interventions act
on potential deaths or failures, making the potential failures the natural
denominator for measuring the effect (leading to a preference for modelling
the standard risk ratio). In contrast, harmful interventions act on potential
survivors or successes (leading to a preference for modelling the survival
ratio).

##  3 A simple causal model

When measuring the effect of medications, a central consideration for choice
of scale is stability: A good effect measure is one that is believed to take
approximately the same value in different groups, given reasonable assumptions
about biological mechanisms. In order to illustrate the theoretical advantages
of Sheps’ approach, we will consider a simplified, highly stylized causal
model that leads to stability of her preferred variant of the relative risk.
While this model is simple, it is consistent with influential work from a
variety of academic traditions, including sufficient component cause (“causal
pie”) models(Rothman, 1976; Huitfeldt et al., 2022).

Consider (unmeasured) individual-level attributes that determine whether a
person responds to treatment. We refer to these attributes as “switches”: if
no switch is present in a person, treatment has no effect on them, whereas
when switches are present in a person, they combine to determine whether and
how treatment affects the outcome. We will refer to combinations of switches
as “switch patterns”; there are four logically possible ways that such switch
patterns can operate, which are shown in Table 1.

Table 1: Switch patterns

Type |  Function |  Equivalent description of function |  Direction of effect |  Example |  Implications when this switch pattern is solely responsible for the effect, and independent of baseline risk

---|---|---|---|---|---
|  |  |  |  |  Expression for prevalence of switch |  Effect measure stability

“Sufficient-causal” |  Treatment is a sufficient cause of the outcome |  Absence of treatment is a necessary cause of absence of the outcome |  Treatment increases risk |  Genetic variant such that carriers get an allergic reaction if they take the drug |  $1-\frac{Pr[Y^{a=1}=0]}{Pr[Y^{a=0}=0]}$ |  $SR$ will be stable between groups with similar prevalence of the switch pattern
---|---|---|---|---|---|---
“Necessary-preventive” |  Treatment is a necessary cause of absence of the outcome |  Absence of treatment is a sufficient cause of the outcome |  Treatment decreases risk |  Bacterial strain such that in patients who are infected with that strain, the outcome event will always occur unless they take a specific drug |  $1-\frac{Pr[Y^{a=0}=0]}{Pr[Y^{a=1}=0]}$ |  $\frac{1}{SR}$ will be stable between groups with similar prevalence of the switch pattern
“Sufficient-preventive” |  Treatment is a sufficient cause of absence of the outcome |  Absence of treatment is a necessary cause of the outcome |  Treatment decreases risk |  Bacterial strain such that in patients who are infected with that strain, the drug always prevents the outcome |  $1-\frac{Pr[Y^{a=1}=1]}{Pr[Y^{a=0}=1]}$ |  $RR$ will be stable between groups with similar prevalence of the switch pattern
“Sufficient-causal” |  Treatment is a necessary cause of the outcome |  Absence of treatment is a sufficient cause of absence of the outcome |  Treatment increases risk |  Genetic variant such that no allergic reactions can occur in the carrier unless they take the drug |  $1-\frac{Pr[Y^{a=0}=1]}{Pr[Y^{a=1}=1]}$ |  $\frac{1}{RR}$ will be stable between groups with similar prevalence of the switch pattern

If the same switch pattern type accounts for treatment response in all
individuals and the switch pattern is independent of baseline risk, then there
will exist some variant of the relative risk whose value, in any group,
reflects the prevalence of the switch pattern, as shown in table 1. When this
is the case, predictors of switch pattern prevalence are effect modifiers on
that scale(Huitfeldt et al., 2019).

Reality is unfortunately rarely that simple: Any reasonable epidemiologist
would certainly be wary about relying on an assumption that only one type of
switch pattern accounts for the effect of treatment. However, in some cases,
one type of switch is predominantly responsible for the effect of treatment,
and in these cases, choosing a model based on the associated variant of the
relative risk may allow heterogeneity to be understood in terms of
biologically interpretable deviation from the most applicable model.

In many specific cases, it is possible to show that the sufficient-causal
pattern is more consistent with how biology is typically understood than the
necessary-causal pattern (leading to a preference for the survival ratio for
exposures that increase incidence, in accordance with Sheps’ recommendation),
and that the sufficient-preventive pattern is more consistent with biology
than the necessary-preventive pattern (leading to a preference for the risk
ratio for exposures that reduce incidence, again in accordance with Sheps’
recommendation). For example, in the setting of a vaccine, a sufficient-
preventive pattern may reflect the fact that the vaccine is sufficient to
prevent the disease in those who are exposed to a particular susceptible
strain of the virus, and has little effect in those who are exposed to a non-
susceptible strain. In contrast, it would be challenging to describe a
biological phenomenon that can be represented as a necessary-preventive
pattern. Similar arguments can be made in many clinically relevant
applications, almost always leading to a preference for the variant of the
relative risk suggested by Sheps. Table 1 provides some additional such
examples.

But if it is true, as a feature of nature, that some switch patterns are much
more common than others, this screams out for an explanation: How did biology
end up being so asymmetric? In one potential attempt to account for this
asymmetry, it may be useful to invoke evolutionary theory. In an environment
where everybody is unexposed, the sufficient-causal and the sufficient-
preventive pattern types are both inert, whereas the necessary-causal and the
necessary-preventive pattern types strongly determine the outcome. Similarly,
in an environment where everybody is exposed, the necessary-causal and
necessary-preventive pattern types are both inert, whereas the sufficient-
causal and sufficient-preventive types strongly determine the outcome.

The switch pattern types that lead to stability of Sheps’ preferred variant of
the relative risk, are exactly those pattern types that would have been inert
in evolutionary history, as modern medical treatments were almost always
unavailable to our ancestors. This hints at a possible explanation for
nature’s asymmetry: Only those switch pattern types that lead to Sheps’
conclusions would have been inert in evolutionary history, and could plausibly
remain in our gene pool as determinants of variation in treatment response.

##  4 A brief history of related ideas

While Sheps was the first to suggest this line of reasoning in medical
statistics, her ideas were anticipated by researchers working in toxicology
and entomology. In a very short paper from 1925, W.S. Abbott, an entomologist
working at the US Department of Agriculture, proposed measuring the effect of
insect sprays using what is now known as Abbott’s Formula(Abbott, 1925). This
formula, which is still used by entomologists today, is equivalent to Sheps’
suggestion for the case where the intervention increases risk of the outcome.

Abbott did not consider the case where exposure reduces risk of the outcome,
which is not surprising because he studied insecticides. That consideration
was left to another another entomologist, albeit one who played a significant
role in the early modern history of statistics: many readers will recognize
C.I. Bliss as the originator of the Probit Model. In 1939, he extended
Abbott’s formula to the setting where exposure reduces incidence, preserving
its equivalence with Sheps’ approach. To do so, he developed the Joint
Independent Action model(Bliss, 1939) , a simple mechanism for drug effect
which has become central to how toxicologists think about interactions between
poisons.

Attempts to introduce these approaches in epidemiology have been made by
several researchers whose research takes place at the intersection between
toxicology and epidemiology. For example, Clarice Weinberg showed that when a
drug acts according to Abbott’s model for the mechanism of action, generalized
linear models should use a log link when considering outcomes whose incidence
are reduced by the drug, and a complementary log link when considering
outcomes whose incidence are reduced by the drug(Weinberg, 1986, 2007, 2012).
This modelling strategy leads to estimating main effects that are consistent
with Sheps’ recommendations. However, it is rarely used in practice, possibly
because it would require data analysts to know the direction of the effect
before specifying the model.

A solution to this problem was provided by van der Laan et al. (2007), who
proposed the switch relative risk, an effect parameter which automatically
selects a variant of the relative risk in accordance with Sheps’ principles.
Independently, Baker and Jackson (2018) proposed the generalized relative risk
reduction (GRRR), a closely related effect measure which encodes all the
relevant information on the interval [-1,1].

Models that imply stability of Sheps’ preferred variant of the relative risk
are also influential in the philosophy and psychology literatures, in the form
of Patricia Cheng’s Power PC theory(Cheng, 1997). It has been argued on
theoretical and empirical grounds that human decision makers use (and should
use) these models to carry causal information from one context to
another(Liljeholm and Cheng, 2007), i.e. for purposes of generalizability.
Clark Glymour, who is a leading philosopher of science, referred to the Power
PC model as a ”brilliant piece of mathematical metaphysics”(Glymour, 2001).

##  5 Conclusions

Among the scales that can be used to measure the effects of medical
interventions, Sheps’ approach is unique in that stability of her preferred
variant of the relative risk may result from well-understood and plausible
biological mechanisms. The underlying ideas have been independently
rediscovered multiple times, by researchers from academic traditions as
diverse as entomology, toxicology, psychology, philosophy, computer science
and epidemiology. The diversity of thinkers who have arrived independently at
the same conclusions, suggests that Sheps’ ideas represent an attractor in
idea space, and that they will continue to be rediscovered as long as her
groundbreaking work remains relatively unknown and unused in practice.

## References

  * Abbott (1925) Abbott, W. S. (1925) A Method of Computing the Effectiveness of an Insecticide.  Journal of Economic Entomology, 18, 265–267.  URL: https://doi.org/10.1093/jee/18.2.265a.
  * Baker and Jackson (2018) Baker, R. and Jackson, D. (2018) A new measure of treatment effect for random-effects meta-analysis of comparative binary outcome data.  arXiv:1806.03471.
  * Bliss (1939) Bliss, C. I. (1939) The Toxicity of Poisons Applied Jointly.  Annals of Applied Biology, 26, 585–615.  URL: https://onlinelibrary.wiley.com/doi/abs/10.1111/j.1744-7348.1939.tb06990.x.  _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1111/j.1744-7348.1939.tb06990.x.
  * Bouckaert and Mouchart (2001) Bouckaert, A. and Mouchart, M. (2001) Sure outcomes of random events: a model for clinical trials.  Statistics in Medicine, 20, 521–543.  URL: https://doi.org/10.1002/sim.659.  Publisher: John Wiley & Sons, Ltd.
  * Cheng (1997) Cheng, P. (1997) From Covariation to Causation: A Causal Power Theory.  Psychological Review, 104, 367–405.
  * Cox (1972) Cox, D. R. (1972) The Analysis of Multivariate Binary Data.  Applied Statistics, 113–120.  URL: http://www.jstor.org/stable/2346482\npapers://28b4d2b8-358f-48a3-a2ff-2ddb6af66501/Paper/p4042.
  * Deeks et al. (2020) Deeks, J., Higgins, J. and Altman (editors), D. (2020) Chapter 10: Analysing data and undertaking meta-analyses.  In Cochrane Handbook for Systematic Reviews of Interventions version 6.1. Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane.  URL: www.training.cochrane.org/handbook.
  * Glymour (2001) Glymour, C. (2001) The mind’s arrows: Bayes nets and graphical causal models in psychology.  The mind’s arrows: Bayes nets and graphical causal models in psychology. Cambridge, MA, US: The MIT Press.  Pages: xv, 222.
  * Greenland and Robins (1986) Greenland, S. and Robins, J. M. (1986) Identifiability, Exchangeability, and Epidemiological Confounding.  International journal of epidemiology, 15, 413–9.  URL: http://www.ncbi.nlm.nih.gov/pubmed/3771081.
  * Guyatt et al. (2013) Guyatt, G. H., Oxman, A. D., Santesso, N., Helfand, M., Vist, G., Kunz, R., Brozek, J., Norris, S., Meerpohl, J., Djulbegovic, B., Alonso-Coello, P., Post, P. N., Busse, J. W., Glasziou, P., Christensen, R. and Schünemann, H. J. (2013) GRADE guidelines: 12. Preparing Summary of Findings tables—binary outcomes.  Journal of Clinical Epidemiology, 66, 158–172.  URL: https://www.jclinepi.com/article/S0895-4356(12)00032-7/fulltext.  Publisher: Elsevier.
  * Huitfeldt et al. (2022) Huitfeldt, A., Fox, M. P., Murray, E. J., Hróbjartsson, A. and Daniel, R. M. (2022) Shall we count the living or the dead?  URL: http://arxiv.org/abs/2106.06316.  ArXiv:2106.06316 [stat].
  * Huitfeldt et al. (2018) Huitfeldt, A., Goldstein, A. and Swanson, S. A. (2018) The choice of effect measure for binary outcomes: Introducing counterfactual outcome state transition parameters.  Epidemiologic Methods, 7.  URL: https://www.degruyter.com/view/j/em.2018.7.issue-1/em-2016-0014/em-2016-0014.xml?format=INT.
  * Huitfeldt et al. (2019) Huitfeldt, A., Swanson, S. A., Stensrud, M. J. and Suzuki, E. (2019) Effect heterogeneity and variable selection for standardizing causal effects to a target population.  European Journal of Epidemiology, 34, 1119–1129.  URL: https://doi.org/10.1007/s10654-019-00571-w.
  * van der Laan et al. (2007) van der Laan, M. J., Hubbard, A. and Jewell, N. P. (2007) Estimation of treatment effects in randomized trials with non-compliance and a dichotomous outcome.  Journal of the Royal Statistical Society: Series B (Statistical Methodology), 69, 463–482.  URL: https://rss.onlinelibrary.wiley.com/doi/abs/10.1111/j.1467-9868.2007.00598.x.
  * Liljeholm and Cheng (2007) Liljeholm, M. and Cheng, P. W. (2007) When Is a Cause the “Same”?: Coherent Generalization Across Contexts.  Psychological Science, 18, 1014–1021.  URL: https://doi.org/10.1111/j.1467-9280.2007.02017.x.  Publisher: SAGE Publications Inc.
  * Rothman (1976) Rothman, K. J. (1976) Causes.  American journal of epidemiology, 104, 587–592.
  * Sackett et al. (1996) Sackett, D. L., Deeks, J. J. and Altman, D. G. (1996) Down with odds ratios!  BMJ Evidence-Based Medicine, 1, 164–166.  URL: https://ebm.bmj.com/content/1/6/164.  Publisher: Royal Society of Medicine Section: EBM Notebook.
  * Sheps (1958) Sheps, M. C. (1958) Shall We Count the Living or the Dead?  The New England Journal of Medicine, 259, 1210–4.  URL: http://www.ncbi.nlm.nih.gov/pubmed/13622912.
  * Sheps (1959) — (1959) An Examination of Some Methods of Comparing Several Rates or Proportions.  Biometrics, 15, 87.  URL: https://www.jstor.org/stable/2527603?origin=crossref.
  * Walter (1998) Walter, S. D. (1998) Odds ratios revisited.  Evidence-Based Medicine.
  * Weinberg (1986) Weinberg, C. R. (1986) Applicability of the simple independent action model to epidemiologic studies involving two factors and a dichotomous outcome.  American Journal of Epidemiology, 123, 162–173.  URL: https://doi.org/10.1093/oxfordjournals.aje.a114211.
  * Weinberg (2007) — (2007) Can DAGs Clarify Effect Modification?  Epidemiology (Cambridge, Mass.), 18, 569–572.  URL: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2235194/.
  * Weinberg (2012) — (2012) Interaction and Exposure Modification: Are We Asking the Right Questions?  American Journal of Epidemiology, 175, 602–605.  URL: https://doi.org/10.1093/aje/kwr495.


* Hermann et al. (2020) B. Hermann, S. Winter, and J. Siegmund. 2020.  Community Expectations for Research Artifacts and Evaluation Processes. In _Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering_ _(ESEC/FSE ’20)_. 469–480.
  * Heumüller et al. (2020) R. Heumüller, S. Nielebock, J. Krüger, and F. Ortmeier. 2020.  Publish or Perish, but do not Forget your Software Artifacts.  _Empirical Software Engineering_ 25, 6 (2020), 4585–4616.
  * Ji et al. (2014) S. Ji, W. Li, M. Srivatsa, and R. Beyah. 2014\.  Structural data de-anonymization: Quantification, practice, and implications. In _Conference on Computer and Communications Security_ _(SIGSAC ’14)_. 1040–1053.
  * Jimenez et al. (2017) I. Jimenez, M. Sevilla, N. Watkins, C. Maltzahn, J. Lofstead, K. Mohror, A. Arpaci-Dusseau, and R. Arpaci-Dusseau. 2017.  The Popper convention: Making reproducible systems evaluation practical. In _International Parallel and Distributed Processing Symposium Workshops_ _(IPDPSW ’17)_. 1561–1570.
  * Johns (1993) G. Johns. 1993.  Constraints on the adoption of psychology-based personnel practices: Lessons from organizational innovation.  _Personnel Psychology_ 46, 3 (1993), 569–592.
  * Kitchenham (2008) B. Kitchenham. 2008\.  The role of replications in empirical software engineering—a word of warning.  _Empirical Software Engineering_ 13 (04 2008), 219–221.
  * Kitto et al. (2008) S. C. Kitto, J. Chesters, and C. Grbich. 2008.  Quality in qualitative research.  _Medical Journal of Australia_ 188, 4 (2008), 243–246.
  * Kitware (2021) Kitware. 2021\.  CMake.  https://cmake.org [Date Accessed: March 13 2021].
  * Kluyver et al. (2016) T. Kluyver, B. Ragan-Kelley, F. Pérez, B. Granger, M. Bussonnier, J. Frederic, K. Kelley, J. Hamrick, J. Grout, S. Corlay, P. Ivanov, D. Avila, S. Abdalla, C. Willing, and Jupyter Development Team. 2016.  Jupyter Notebooks - a publishing format for reproducible computational workflows. In _Positioning and Power in Academic Publishing: Players, Agents and Agendas_. IOS Press, 87–90.
  * Kotti et al. (2020) Z. Kotti, K. Kravvaritis, K. Dritsa, and D. Spinellis. 2020.  Standing on shoulders or feet? An extended study on the usage of the MSR data papers.  _Empirical Software Engineering_ 25, 5 (2020), 3288–3322.
  * Krishnamurthi (2013a) S. Krishnamurthi. 2013a.  Artifact Evaluation for Software Conferences.  _SIGSOFT Software Engineering Notes_ 38, 3 (May 2013), 7–10.
  * Krishnamurthi (2013b) S. Krishnamurthi. 2013b.  Examining “Reproducibility in Computer Science”.  http://cs.brown.edu/~sk/Memos/Examining-Reproducibility/ [Date Accessed: January 6th 2020].
  * Krishnamurthi (2014) S. Krishnamurthi. 2014\.  Guidelines for Packaging AEC Submissions.  https://www.artifact-eval.org/guidelines.html [Date Accessed: July 17th 2020].
  * Krishnamurthi and Vitek (2015) S. Krishnamurthi and J. Vitek. 2015.  The Real Software Crisis: Repeatability As a Core Value.  _Commun. ACM_ 58, 3 (Feb. 2015), 34–36.
  * Li-Thiao-Té (2012) S. Li-Thiao-Té. 2012\.  Literate program execution for reproducible research and executable papers.  _Procedia Computer Science_ 9 (2012), 439–448.
  * Lindvall et al. (2005) M. Lindvall, I. Rus, F. Shull, M. Zelkowitz, P. Donzelli, A. Memon, V. Basili, P. Costa, R. Tvedt, L. Hochstein, S. Asgari, C. Ackermann, and D. Pech. 2005\.  An evolutionary testbed for software technology evaluation.  _Innovations in Systems and Software Engineering_ 1, 1 (01 Apr 2005), 3–11.
  * Meng et al. (2016) H. Meng, D. Thain, A. Vyushkov, M. Wolf, and A. Woodard. 2016.  Conducting reproducible research with Umbrella: Tracking, creating, and preserving execution environments. In _International Conference on e-Science_. 91–100.
  * Miranda et al. (2014) M. Miranda, R. Ferreira, C. R. B. de Souza, F. M. F. Filho, and L. Singer. 2014.  An exploratory study of the adoption of mobile development platforms by software engineers. In _International Conference on Mobile Software Engineering and Systems_ _(MOBILESoft ’14)_. 50–53.
  * Morse (1991) J. M. Morse. 1991\.  Qualitative nursing research: A free-for-all.  _Qualitative nursing research: A contemporary dialogue_ (1991), 14–22.
  * Murphy-Hill et al. (2019) E. R. Murphy-Hill, E. K. Smith, C. Sadowski, C. Jaspan, C. Winter, M. Jorde, A. Knight, A. Trenk, and S. Gross. 2019\.  Do developers discover new tools on the toilet?. In _International Conference on Software Engineering_ _(ICSE ’19)_. 465–475.
  * Narayanan and Shmatikov (2008) A. Narayanan and V. Shmatikov. 2008.  Robust de-anonymization of large sparse datasets. In _Symposium on Security and Privacy_ _(SP ’08)_. 111–125.
  * NIH (2003) NIH. 2003.  NIH Data Sharing Policy and Implementation Guidance.  https://grants.nih.gov/grants/policy/data_sharing/data_sharing_guidance.htm [Date Accessed: July 21st 2020].
  * NSF (2011) NSF. 2011.  Dissemination and Sharing of Research Results.  https://www.nsf.gov/bfa/dias/policy/dmp.jsp [Date Accessed: July 21st 2020].
  * O’Neill et al. (1998) H. M. O’Neill, R. W. Pouder, and A. K. Buchholtz. 1998\.  Patterns in the diffusion of strategies across organizations: Insights from the innovation diffusion literature.  _Academy of Management Review_ 23, 1 (1998), 98–114.
  * Oracle (2021) Oracle. 2021.  VirtualBox.  https://www.virtualbox.org [Date Accessed: March 13 2021].
  * Patterson et al. (1999) D. Patterson, L. Snyder, and J. Ullman. 1999.  Evaluating Computer Scientists and Engineers For Promotion and Tenure.  _Computing Research News_ (August 1999).
  * Premkumar et al. (1994) G. Premkumar, K. Ramamurthy, and S. Nilakanta. 1994\.  Implementation of electronic data interchange: an innovation diffusion perspective.  _Journal of Management Information Systems_ 11, 2 (1994), 157–186.
  * PyPA (2021) PyPA. 2021.  pip – The Python Package Installer.  https://pip.pypa.io/en/stable [Date Accessed: March 13 2021].
  * QEMU (2021) QEMU. 2021.  QEMU.  https://www.qemu.org [Date Accessed: March 13 2021].
  * RedHat (2021) RedHat. 2021.  Podman.  https://github.com/containers/podman [Date Accessed: March 13 2021].
  * Rogers (2010) E. M. Rogers. 2010\.  _Diffusion of innovations_.  Simon and Schuster.
  * Rougier et al. (2017) N. P. Rougier, K. Hinsen, F. Alexandre, T. Arildsen, L. A. Barba, F. C. Y. Benureau, C. Titus Brown, P. de Buyl, O. Caglayan, A. P. Davison, M. Delsuc, G. Detorakis, A. K. Diem, D. Drix, P. Enel, B. Girard, O. Guest, M. G. Hall, R. N. Henriques, X. Hinaut, K. S. Jaron, M. Khamassi, A. Klein, T. Manninen, P. Marchesi, D. McGlinn, C. Metzner, O. L. Petchey, H. E. Plesser, T. Poisot, K. Ram, Y. Ram, E. B. Roesch, C. Rossant, V. Rostami, A. Shifman, J. Stachelek, M. Stimberg, F. Stollmeier, F. Vaggi, G. Viejo, J. Vitay, A. E. Vostinar, R. Yurchak, and T. Zito. 2017.  Sustainable computational science: the ReScience initiative.  _PeerJ Computer Science_ 3 (2017), 1–8.  Issue 12.
  * Saldaña (2015) J. Saldaña. 2015\.  _The coding manual for qualitative researchers_.  Sage.
  * Shull et al. (2002) F. Shull, V. Basili, J. Carver, J. C. Maldonado, G. H. Travassos, M. Mendonca, and S. Fabbri. 2002.  Replicating software engineering experiments: addressing the tacit knowledge problem. In _International Symposium on Empirical Software Engineering_ _(ESEM ’02)_. 7–16.
  * Shull et al. (2008) F. J. Shull, J. C. Carver, S. Vegas, and N. Juristo. 2008\.  The role of replications in Empirical Software Engineering.  _Empirical Software Engineering_ 13, 2 (01 Apr 2008), 211–218.
  * Silverman and Marvasti (2008) D. Silverman and A. Marvasti. 2008.  _Doing qualitative research: A comprehensive guide_.  Sage.
  * Smith et al. (2016) A. M. Smith, D. S. Katz, and K. E. Niemeyer. 2016.  Software Citation Principles.  _PeerJ Computer Science_ 2 (2016), e86.
  * Stodden (2009a) V. Stodden. 2009a.  Enabling Reproducible Research: Open Licensing for Scientific Innovation.  _International Journal of Communications Law and Policy_ 13 (2009).
  * Stodden (2009b) V. Stodden. 2009b.  The Legal Framework for Reproducible Scientific Research: Licensing and Copyright.  _Computing in Science & Engineering_ 11, 1 (2009), 35–40.
  * Stodden et al. (2014) V. Stodden, F. Leisch, and R. D. Peng. 2014.  _Implementing reproducible research_.  CRC Press.
  * Teshima et al. (2013) C. G. Teshima, B. Cleary, and L. Singer. 2013.  A study of innovation diffusion through link sharing on stack overflow. In _Working Conference on Mining Software Repositories_ _(MSR ’13)_. 81–84.
  * Timperley et al. (2018) C. S. Timperley, S. Stepney, and C. Le Goues. 2018\.  BugZoo: A platform for studying software bugs. In _International Conference on Software Engineering: Companion Proceeedings_. 446–447.
  * Wolfe (1994) R. A. Wolfe. 1994\.  Organizational innovation: Review, critique and suggested research directions.  _Journal of Management Studies_ 31, 3 (1994), 405–431.
  * Wright et al. (1995) R. E. Wright, J. C. Palmar, and D. C. Kavanaugh. 1995\.  The importance of promoting stakeholder acceptance of educational innovations.  _Education_ 115, 4 (1995), 628–633.
  * Zerhouni (2003) E. Zerhouni. 2003\.  The NIH roadmap.

  *[DoI]: Diffusion of Innovations
  *[IS]: Implementation Science
  *[ICSE]: the International Conference on Software Engineering
  *[FSE]: the Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering
  *[ASE]: the International Conference on Automated Software Engineering
  *[EMSE]: Empirical Software Engineering
  *[ACM]: Association for Computing Machinery
  *[AEC]: Artifact Evaluation Committee
  *[AECs]: Artifact Evaluation Committee
  *[IP]: intellectual property
  *[CRA]: Computing Research Association
  *[NSF]: National Science Foundation
  *[NIH]: National Institutes of Health
  *[EPSRC]: Engineering and Physical Sciences Research Council
  *[EC]: European Commission
  *[MSR]: the International Conference on Mining Software Repositories