# Some Statistical and Data Challenges When Building Early-Stage Digital
Experimentation and Measurement Capabilities

C. H. Bryan Liu

###### Abstract

Digital experimentation and measurement (DEM) capabilities – the knowledge and
tools necessary to run experiments with digital products, services, or
experiences and measure their impact – are fast becoming part of the standard
toolkit of digital/data-driven organisations in guiding business decisions.
Many large technology companies report having mature DEM capabilities, and
several businesses have been established purely to manage experiments for
others. Given the growing evidence that data-driven organisations tend to
outperform their non-data-driven counterparts, there has never been a greater
need for organisations to build/acquire DEM capabilities to thrive in the
current digital era.

This thesis presents several novel approaches to statistical and data
challenges for organisations building DEM capabilities. We focus on the
fundamentals associated with building DEM capabilities, which lead to a richer
understanding of the underlying assumptions and thus enable us to develop more
appropriate capabilities. We address _why_ one should engage in DEM by
quantifying the benefits and risks of acquiring DEM capabilities. This is done
using a ranking under lower uncertainty model, enabling one to construct a
business case. We also examine _what_ ingredients are necessary to run digital
experiments. In addition to clarifying the existing literature around
statistical tests, datasets, and methods in experimental design and causal
inference, we construct an additional dataset and detailed case studies on
applying state-of-the-art methods. Finally, we investigate _when_ a digital
experiment design would outperform another, leading to an evaluation framework
that compares competing designs’ data efficiency.

These approaches aim to enable one to run experiments that produce less biased
estimates more quickly and adapt to various business constraints. As we
maintain the theoretical rigour, we also emphasise applied use cases,
interpretable processes/results, and practical tradeoffs, all to ensure that
the contributions are accessible to researchers and practitioners from diverse
scientific backgrounds.

I confirm that the thesis reports original research work done by myself during
the programme of study. The thesis has not been submitted elsewhere for
examination for a PhD degree. Part of the thesis is adapted from research
papers that I first-authored and published during the programme of study. They
are listed in Section 1.4 and stated at the beginning of relevant chapters
where applicable. In addition, I have appropriately referenced ideas from
others in the thesis and sought the necessary permission to use copyrighted
materials.

The thesis is written at a time when large language models (LLMs) are
sufficiently advanced to produce ”human-like” writing on an arbitrary topic
and refine the text based on external prompts. I see much potential with such
technology in research and industry practice. That said, the use of LLMs in
academic writing remains highly controversial as of 2023, with many unresolved
issues on correctness, academic integrity and plagiarism, and publishing
ethics [2].

As such, I also confirm that I did not use any LLMs or services derived from
them to write this thesis. That said, I used (the non-LLM part of) Grammarly,
an online contextual proofreading platform, to help correct the written text’s
spelling, punctuation, and grammar. The platform also flags redundant,
overused, or informal phrases and sentences based on set rules to help improve
the overall delivery of the thesis. However, the decision to incorporate the
suggestions solely rests with me.

The copyright of this thesis rests with the author. Unless otherwise
indicated, its contents are licensed under a Creative Commons Attribution 4.0
International Licence (CC BY 4.0).

Under this licence, you may copy and redistribute the material in any medium
or format for both commercial and non-commercial purposes. You may also create
and distribute modified versions of the work. This on the condition that you
credit the author.

When reusing or sharing this work, ensure you make the licence terms clear to
others by naming the licence and linking to the licence text. Where a work has
been adapted, you should indicate that the work has been changed and describe
those changes.

Please seek permission from the copyright holder for uses of this work that
are not included in this licence or permitted under UK Copyright Law.

Acknowledgements

Academic research is never a solo effort, and I would like to acknowledge the
many people who have helped, in various ways, for me to deliver this thesis.

First and foremost, I would like to thank my supervisor, Prof Emma McCoy, for
her consistent support and guidance throughout the years. She has suggested
many worthy research directions, challenged many potential ideas, and helped
clarify my exposition on many topics. I would also like to thank Dr Seth
Flaxman and Prof Robin Evans for the initial project idea discussions, as well
as Prof Niall Adams, Dr Ciara Pike-Burke, Prof Nick Heard, and the anonymous
reviewers for their feedback during the initial stages of the PhD.

I am fortunate to have met many collaborators whom I enjoy the many
intellectually stimulating discussions and opportunities to publicise the
latest results in high-impact venues with: Dr Elaine Bettaney, Dr Ângelo
Cardoso, Dr Benjamin Chamberlain, Paul Couturier, Prof Marc Deisenroth, Georg
Grob, Dr Duncan Little, and Dr Roberto Pagliari. I have also met many bright
minds at Imperial College London and the University of Oxford, many affiliated
with the EPSRC CDT in Modern Statistics and Statistical Machine Learning
(StatML CDT), whose conversations provided me with new perspectives on the
challenges at hand and enabled me to discover new opportunities.

The thesis is one of the many outputs from a fruitful academic-industry
collaboration between the StatML CDT at Imperial and Oxford and the AI and
Data Science Platform at ASOS.com, with myself also employed at the latter as
a data/machine learning scientist. I am blessed to have also worked with and
learnt from many talented colleagues at ASOS, including my managers Dr
Benjamin Chamberlain, Papinder Dosanjh, and Dawn Rollocks; my teammates in the
Customer Understanding (2016–18), Experimentation (2018–19), Prophecy
(2019–20), Cassandra (2020), Kernel (2021–22), Experimentation (2022), and the
Alchemists (2022–23) teams; and many more unnamed. Among other things, they
have ensured that my research remains relevant to industry applications. The
research is also generously funded by the two organisations mentioned above
(EPSRC grant no. EP/S023151/1), without which I probably would not have taken
on the PhD programme at all.

Finally, I would like to thank my friends and family, especially my mum
Yvonne, my dad Haston, my sister Nicole, and my grandmother Cecilia, for their
love and warm encouragement while I took on the ambitious task of studying a
PhD while holding an industry job. My PhD years were dominated by the COVID-19
pandemic, and travel restrictions have made visiting them virtually impossible
for a few years. Fortunately, the availability of economical video calls has
made the isolation much more bearable.

> “The true method of knowledge is experiment.”
>
> – William Blake
>
>
> “Negative results are just what I want. They’re just as valuable to me as
> positive results. I can never find the thing that does the job best until I
> find the ones that don’t.”
>
> – Thomas A. Edison
>
>
> “An experiment is a question which science poses to Nature and a measurement
> is the recording of Nature’s answer.”
>
> – Max Planck
>
>
> “Measurement is the first step that leads to control and eventually to
> improvement. If you can’t measure something, you can’t understand it. If you
> can’t understand it, you can’t control it. If you can’t control it, you
> can’t improve it.”
>
> – H. James Harrington
>

###### Contents

  1. 1 Introduction
    1. 1.1 Motivation
    2. 1.2 Key Research Questions
    3. 1.3 Contributions and Chapter Organisation
    4. 1.4 Publications
    5. 1.5 List of Acronyms and Mathematical Symbols
  2. 2 What is the Value of Digital Experimentation and Measurement Capabilities?
    1. 2.1 Motivation
    2. 2.2 Related work
    3. 2.3 Mathematical Formulation
      1. 2.3.1 Modelling values with statistical distributions
      2. 2.3.2 Key results
    4. 2.4 Calculating the Expectation
      1. 2.4.1 Expectation under normal assumptions
    5. 2.5 Calculating the Variance
      1. 2.5.1 $\textrm{Var}(\mathcal{H}_{(r)})$/$\textrm{Var}(\mathcal{L}_{(s)})$
      2. 2.5.2 $\textrm{Var}(\mathcal{V}_{\mathcal{I}(r)})$/$\textrm{Var}(\mathcal{V}_{\mathcal{J}(s)})$
      3. 2.5.3 $\textrm{Var}(\mathcal{W}_{1})$/$\textrm{Var}(\mathcal{W}_{2})$
      4. 2.5.4 $\textrm{Var}(\mathcal{D})$
    6. 2.6 Empirical Verification
    7. 2.7 Case Study
      1. 2.7.1 e-Commerce companies
      2. 2.7.2 Marketing companies
    8. 2.8 Empirical Extensions
      1. 2.8.1 Valuation under independent $t$-distributed assumptions
      2. 2.8.2 Partial estimation/measurement noise reduction
    9. 2.9 A Brief Recap
  3. 3 Statistical Testing
    1. 3.1 Motivation
    2. 3.2 Pre-matter and Considerations
    3. 3.3 Alternative Approaches to Formal Tests
      1. 3.3.1 The inter-ocular trauma test
      2. 3.3.2 Effect size
      3. 3.3.3 Confidence Intervals
    4. 3.4 Null Hypothesis Significance Tests
    5. 3.5 Examples of NHSTs
      1. 3.5.1 $z$-test
      2. 3.5.2 $t$-tests
      3. 3.5.3 The binomial test of proportions
    6. 3.6 Key Quantities in NHST
      1. 3.6.1 $p$-value
      2. 3.6.2 Type I and II errors
      3. 3.6.3 Significance level
      4. 3.6.4 Power
      5. 3.6.5 Required sample size
      6. 3.6.6 Minimum detectable effect
    7. 3.7 Non-parametric Tests
      1. 3.7.1 Mann-Whitney $U$ test
      2. 3.7.2 $\chi^{2}$ goodness-of-fit test
    8. 3.8 Sequential Tests
    9. 3.9 Bayesian Tests
    10. 3.10 A Brief Recap
  4. 4 Datasets for Digital Experiments
    1. 4.1 Motivation
    2. 4.2 A Taxonomy for Digital Experiment Datasets
    3. 4.3 Public Online Controlled Experiment Datasets
    4. 4.4 Matching Dataset Taxonomy with Statistical Tests
      1. 4.4.1 Effect size and Welch’s $t$-test
      2. 4.4.2 Experiments with adaptive stopping
      3. 4.4.3 Non-parametric tests
    5. 4.5 A Novel Dataset for Experiments with Adaptive Stopping
      1. 4.5.1 Potential use cases
      2. 4.5.2 Ethical considerations
    6. 4.6 A Brief Recap
  5. 5 Recent Advances in Digital Experimentation Methods
    1. 5.1 Motivation
    2. 5.2 From Correlation to Causation – The Potential Outcomes Framework
    3. 5.3 Classes and Examples of Digital Experiments
      1. 5.3.1 Randomised controlled trials
      2. 5.3.2 Quasi-experiments
      3. 5.3.3 Natural experiments
      4. 5.3.4 Special case – Experiments for personalisation strategies
    4. 5.4 Advances in Online Controlled Experiments
      1. 5.4.1 Advanced statistical methods
      2. 5.4.2 Other topics
    5. 5.5 A Spotlight on Dependent Responses in e-Commerce Experiments
      1. 5.5.1 Beware of Hidden Inflation in Measurement Uncertainty
      2. 5.5.2 Re-estimating the SE using bootstrap
      3. 5.5.3 Impact on OCE decisions
      4. 5.5.4 Lessons Learned During Deployment
    6. 5.6 Advances in Quasi-experiments and Natural Experiments
      1. 5.6.1 CATE and Conditional Exchangeability Revisited
      2. 5.6.2 Methods and Examples
      3. 5.6.3 A mini spotlight on geo-experiments
    7. 5.7 A Brief Recap
  6. 6 An Evaluation Framework for Personalisation Strategy Experiment Designs
    1. 6.1 Motivation
    2. 6.2 Evaluation Framework
      1. 6.2.1 User grouping
      2. 6.2.2 Experiment setups
      3. 6.2.3 Evaluation criteria
    3. 6.3 Comparing Experiment Setups
      1. 6.3.1 Actual and minimum detectable effect sizes
      2. 6.3.2 Is dilution always bad?
      3. 6.3.3 When is dual control more effective?
    4. 6.4 Empirical Verification
    5. 6.5 A Brief Recap
  7. 7 Conclusion
    1. 7.1 Summary of Thesis Achievements
    2. 7.2 Future Work
  8. A Miscellaneous Mathematical Results
    1. A.1 Probability in Generating Specific Order Statistic Pairs
      1. A.1.1 Representing the probability as a certain number of successes in multiple Bernoulli trials
      2. A.1.2 Modelling the success count distribution using beta-binomials
      3. A.1.3 Estimation under normal assumptions
      4. A.1.4 Some basic validations
    2. A.2 Calculations Related to Dilution in Personalisation Strategy Experiment Designs
      1. A.2.1 The first criterion
      2. A.2.2 The second criterion
    3. A.3 Calculations Related to Dual Control in Personalisation Strategy Experiment Designs
      1. A.3.1 The master inequality
      2. A.3.2 Simplifying RHS by assuming similar response variances
      3. A.3.3 Simplifying LHS by assuming similar sample sizes
      4. A.3.4 Required number of users for dual control to emerge superior

###### List of Tables

  1. 1.1 List of mathematical symbols and the quantity/concept represented by them.
  2. 2.1 The number of statistical tests with the centred 95% confidence interval containing the derived theoretical value for each quantity of interest.
  3. 4.1 Results from the first ever survey of OCE datasets.
  4. 4.2 Comparing the number of reject / not reject verdicts given by Welch’s $t$-test and an mSPRT at the end of an experiment for all four metrics.
  5. 5.1 Pitfalls in running an OCE end-to-end as described in nine selected works.
  6. 5.2 Summary of the three online retail/e-commerce transaction datasets featured in this paper (post data cleaning).
  7. 5.3 The ratio between the bootstrap standard error (SE) estimates and the vanilla sample SE estimates for ASP.
  8. 6.1 All group-scenario combinations in our evaluation framework for personalisation strategy experiments.
  9. 6.2 Number of evaluations where the theoretical value of the quantities falls between the 95% bootstrap confidence interval for each experiment setup.
  10. A.1 Mean Kullback-Leibler (KL) divergence between the empirical marginal distribution of $\mathbb{P}(\mathcal{I}(r)=\mathcal{J}(s))$ and distributions produced from different modelling approaches.

###### List of Figures

  1. 1.1 Illustration of an A/B test.
  2. 2.1 Prioritising projects according to their value under different noise levels in value estimation.
  3. (a) High estimation noise
  4. (b) Low estimation noise
  5. 2.2 The generative model in the ranking under lower uncertainty problem in plate notation.
  6. 2.3 The generative model in the ranking under lower uncertainty problem when two distinct noise levels are involved in plate notation.
  7. 2.4 Relationship between different variances/covariances used to calculate the variance of the value gained when the estimation noise is reduced.
  8. 2.5 Relationship between different quantities in a three-item generative model under the ranking under lower uncertainty problem.
  9. 2.6 Histogram of the theoretical quantity’s percentile rank compared to the empirical samples across multiple statistical tests.
  10. (a) $\mathbb{E}(\mathcal{W})$
  11. (b) $\mathbb{E}(\mathcal{D})$
  12. 2.7 The value gained by having some digital experimentation and measurement capabilities $\mathcal{D}$ under different capacity $M$ in the case study on 6,700 e-commerce experiments reported by Browne and Johnson [30].
  13. (a) $(1\%)^{2},(0.8\%)^{2}$
  14. (b) $(1\%)^{2},(0.6\%)^{2}$
  15. (c) $(1\%)^{2},(0.4\%)^{2}$
  16. (d) $(0.8\%)^{2},(0.6\%)^{2}$
  17. (e) $(0.8\%)^{2},(0.4\%)^{2}$
  18. (f) $(0.6\%)^{2},(0.4\%)^{2}$
  19. 2.8 The value gained by having some digital experimentation and capabilities $\mathcal{D}$ under different capacity $M$ in the case study on 184 marketing experiments reported by Johnson et al. [136].
  20. (a) $(5\%)^{2},(0.8\%)^{2}$
  21. (b) $(5\%)^{2},(0.6\%)^{2}$
  22. (c) $(5\%)^{2},(0.4\%)^{2}$
  23. (d) $(2\%)^{2},(0.8\%)^{2}$
  24. (e) $(2\%)^{2},(0.6\%)^{2}$
  25. (f) $(2\%)^{2},(0.4\%)^{2}$
  26. (g) $(1\%)^{2},(0.8\%)^{2}$
  27. (h) $(1\%)^{2},(0.6\%)^{2}$
  28. (i) $(1\%)^{2},(0.4\%)^{2}$
  29. (j) $(0.8\%)^{2},(0.6\%)^{2}$
  30. (k) $(0.8\%)^{2},(0.4\%)^{2}$
  31. (l) $(0.6\%)^{2},(0.4\%)^{2}$
  32. 2.9 The value gained by having some some digital experimentation and measurement capabilities under normal assumptions and $t$-distributed assumptions.
  33. (a) e-Commmerce - $(1\%)^{2},(0.4\%)^{2}$
  34. (b) e-Commerce - $(0.6\%)^{2},(0.4\%)^{2}$
  35. (c) Marketing - $(2\%)^{2},(0.4\%)^{2}$
  36. (d) Marketing - $(0.8\%)^{2},(0.4\%)^{2}$
  37. 2.10 The near-linear relationship between the proportion of business propositions with their value estimated under a lower estimation noise and the improvement in mean true value of the selected propositions under normal assumptions.
  38. (a) $N=50$, $M=5$
  39. (b) $N=50$, $M=20$
  40. (c) $N=2500$, $M=250$
  41. (d) $N=2500$, $M=1000$
  42. (e) $N=50$, $M=5$
  43. (f) $N=50$, $M=20$
  44. (g) $N=2500$, $M=250$
  45. (h) $N=2500$, $M=1000$
  46. 4.1 Distribution of $p$-values attained by the 99 OCEs in the ASOS Digital Experiment Dataset using Welch’s $t$-tests, split by decision metrics.
  47. 4.2 Change in $p$-values in a mixed sequential probability ratio test and posterior belief in the null hypothesis in a Bayesian hypothesis test during the experiment for five experiments selected at random.
  48. 5.1 The onion model for different types of experiments.
  49. 5.2 Kernel density estimation plots showing a user’s transaction value/size against that of their next transaction in three online retail transaction datasets.
  50. 5.3 The ratio between the one-way bootstrap standard error (SE) estimates and the vanilla sample SE estimates for different business metrics.
  51. 5.4 Trajectories of the one-way bootstrap standard error (SE) estimates and the vanilla SE estimates under different dataset-metric combinations.
  52. 5.5 The actual test power of and the coverage of the centred confidence interval from a two-tailed $z$-test against different standard error ratios.
  53. 6.1 Venn diagram of the user groups in our evaluation framework for personalisation strategy experiment designs.
  54. 6.2 Experiment setups overlaid on the user grouping Venn diagram in Figure 6.1.
  55. A.1 Figure 2.3 and RHS of Figure 2.5 restated.
  56. A.2 Heatmap showing the empirical distribution of $\mathbb{P}(\mathcal{I}(r)=\mathcal{J}(s))$ across different $r$ and $s$.
  57. A.3 The marginal distribution of $\mathbb{P}(\mathcal{I}(r)=\mathcal{J}(s))$ across different $s$ for a given $r$.
  58. A.4 The fitted marginal distributions overlaid on Figure A.3.
  59. A.5 The fitted marginal distributions overlaid on the empirical marginal distributions of $\mathbb{P}(\mathcal{I}(r)=\mathcal{J}(s))$ ($N=10$).

##  Chapter 1 Introduction

###  1.1 Motivation

The value of making data-driven or data-informed decisions has become
increasingly clear in recent years [187, 256]. Key to making data-driven
decisions is the ability to accurately measure a given choice’s impact and
experiment with possible alternatives. We define digital experimentation and
measurement (DEM) capabilities as the knowledge and tools necessary to run
experiments (controlled or otherwise) with different digital products,
services, or experiences and measure their impact. The capabilities may be an
online controlled experiment framework, a team of analysts, or a system
capable of performing machine learning-aided causal inference.

The simplest example of a digital experiment is what is commonly known as an
A/B test [155]. Suppose we are interested in whether offering free delivery to
users of an e-commerce website will lead to more of them making a purchase
(i.e., an increased conversion rate in business speak). We set up an
experiment where incoming users are randomly split into two groups, where one
group is shown a ”free delivery” banner on the website (the _treatment_),
while the other acts as the _control_ – being shown the original website
without any mention of free delivery (see Figure 1.1). We calculate a
_decision metric_ (here, the conversion rate) for both groups based on
responses from their members and compare the decision metrics using a
statistical test to draw causal statements about the treatment. The approach
is popular in digital organisations, with the largest technology companies
having reported running hundreds or thousands of experiments at any given time
[149, 260, 288].

Figure 1.1: Illustration of an A/B test (a randomised controlled trial with
two parallel groups) designed to test the claim that “offering free delivery
to users of an e-commerce website will lead to a larger proportion of the said
users making a purchase.” Incoming users are split randomly into two groups,
where one group (A) acts as the control, and the other group (B) is shown a
”free delivery” banner on the website as the treatment.

From a statistical point of view, digital experimentation and measurement is
essentially the application of experimental design and causal inference
methods in a digital setting. Likewise, the free delivery experiment mentioned
above is an online randomised controlled trial. Readers will agree that the
underlying experimental design and causal inference methods have been
established and frequently applied in agriculture, medicine, and economics for
decades.

That said, the arrival of the Web and, subsequently, the ability to run an
experiment end-to-end online have brought different opportunities and
challenges. The differences are well documented in [156], which necessitate
the development of a different set of tools and processes exclusively for
digital experiments. In some cases, they also require a complete rethink of
how we approach experiment design and analysis. Under such premises, we
examine the statistical and data challenges one faces when building DEM
capabilities.

##### Guiding Principles

Before we state the research questions in detail (see section below), we
outline the three guiding principles that lead us to the questions. Firstly,
similar to many contemporaries in digital experimentation, we seek methods
that enable one to make better decisions more quickly and thus drive
business/organisational growth. The methods may be those that produce less
biased estimates, generate measurements more quickly, or give alternatives
that can measure the impact of an intervention under various constraints in
practice (e.g., inability to perform user-level random assignment in the “free
delivery” example above).

Secondly, we seek interpretable processes and results alongside performance to
maintain engagement from those beyond our field. We, humans, value the
understanding of why a decision is made as such, and the use of black-box
models or overly complex procedures in decision-making often faces additional
barriers in gaining trust from stakeholders despite its potential to improve
performance [237].

Lastly, and perhaps more importantly, we seek to make the knowledge and tools
involved in digital experimentation accessible, thus ensuring widened
participation.111Clearly, a PhD thesis is a poor medium to engage with a non-
technical audience. We thus limit the scope of “widened participation” to
include researchers and practitioners with diverse academic backgrounds. Many
recent advances in digital experimentation are proposed by researchers and
practitioners who operate in an environment with mature capabilities. Focusing
on those works may give the impression that mature capabilities are the norm
and that we only need a little work to explain the basics on top of what is
already available. Both are not true – the DEM capabilities of many
individuals and organisations are still in their infancy. These individuals
and organisations will benefit from us continuing to consolidate and clarify
the building blocks in digital experiments, e.g., statistical tests, datasets,
and decision metrics.

###  1.2 Key Research Questions

The thesis aims to address digital experimenters’ and organisations’
challenges as they build DEM capabilities. As the thesis title suggests, most
of the work concerns statistical and data challenges one encounters at the
early stage of the process. It also touches upon topics encountered by those
with established capabilities, enabling readers to take away something
wherever they are on the experimentation maturity spectrum [90]. We ask the
following questions:

##### Why should one engage in digital experimentation and measurement?

While running experiment(s) to evaluate a hypothesis is second nature to
scientists, it is unnatural in the business world. Business decision-makers
prefer having business cases, ideally complete with a cost-benefit analysis.
The ability to quantify the benefits of acquiring DEM capabilities (the
knowledge and tools necessary to experiment and measure) will strengthen our
business case. It enables us to speak in the language of the business and thus
stand a better chance of garnering their support.

##### What ingredients do we require to run experiments successfully in a
digital setting?

Committing to investing in DEM capabilities is the first of many steps to reap
the benefits. To run their first experiment successfully, organisations
require, at a minimum, a clear hypothesis and evaluation criteria, detailed
procedures to deliver different treatments to and collect responses from
experiment participants, plus basic knowledge in data processing and
statistical testing. They would also require knowledge of advanced
mathematical and statistical methods, powerful computer systems with
sufficient computing power and data storage capacity, and a growth mindset
organisational culture that embraces scientific, data-driven discoveries to
scale their experiment operations. Each topic, whilst important, warrants
extensive consideration. The thesis will focus on statistical testing, data,
and advanced methods. We will provide a brief overview and pointers to
relevant work for other topics for those interested.

##### When would an experiment design outperform another?

In the parable of two woodcutters, the more experienced woodcutter emerged
victorious in a woodchopping contest by spending most of their time sharpening
their axe. In the same spirit, it is vital to continuously improve our tools,
one of which is experiment designs, as we engage in experimentation efforts.
While there are many qualities that we can use to compare two experiment
designs, in digital experimentation, we usually care about _data efficiency_.
This is represented by the minimum number of experiment participants required
to reach a statistically sound conclusion.222This is closely related to
optimal design [8] and the asymptotic efficiency of a statistical test [254]
in the statistical theory literature. We will approach the challenge from a
more applied point of view. Increasing the data efficiency (i.e., requiring
fewer experiment participants) enables experimenters to make decisions sooner,
leading to better organisational performance as one can deploy improvements
and roll back harmful changes more quickly.

###  1.3 Contributions and Chapter Organisation

The thesis is divided into seven chapters, each detailing contributions to a
different statistical and data topic necessitated by developing early-stage
DEM capabilities. The broad coverage of topics means the thesis embeds the
background knowledge required for each topic within the individual chapters.

The remainder of _Chapter 1_ covers housekeeping items: a list of publications
from the research programme and a list of mathematical symbols and the
quantities/concepts they represent.

_Chapter 2_ introduces the ranking under lower uncertainty problem as a novel
model to value DEM capabilities. This arises from the observation that DEM
capabilities reduce measurement uncertainty on the value of individual
business propositions, improving the inherent value of business propositions
chosen during prioritisation. We derive the expected gain and the variance of
our gain estimate and provide case studies based on large-scale meta-analyses.
This addresses the first research question and enables one to build a
quantitative business case to justify investment in the capabilities.

_Chapter 3_ provides an introduction to statistical testing in digital
experimentation that is both mathematically rigorous and accessible to
researchers and practitioners with different backgrounds. It starts from the
very beginning of a statistical test (i.e., specifying the statistical
hypotheses), provides a detailed treatment on the popular null hypothesis
significance testing (NHST) framework, and involves other more advanced test
paradigms (e.g., sequential and Bayesian testing) and alternatives. It aims to
provide a sufficient theoretical grounding to appreciate what is happening
behind the scenes while highlighting the pitfalls in test design and
interpretation that can trap even the most experienced.

_Chapter 4_ describes a systemic investigation of datasets in digital
experiments. We create the first ever taxonomy for digital experiment datasets
and compile the first ever survey for publicly available online controlled
experiment datasets. We also map the relationship between the dataset taxonomy
and statistical tests discussed in Chapter 3. The taxonomy and survey also
identify gaps in dataset availability, leading to the development of the ASOS
Digital Experiments Dataset. This first real, multi-experiment time series
dataset enables the design and running of experiments with adaptive stopping.

_Chapter 5_ reviews existing challenges in digital experimentation and the
established and emerging methods that address such challenges. We also
complement the review by viewing some of the reported methodological advances
through a practical lens. This is done by interleaving the review with case
studies on randomised controlled trials with dependent responses and quasi-
experiments with geographical regions as treatment units. They provide
evidence on the extent of reported challenges and practical considerations
when implementing relevant methods. Together with Chapters 3 and 4, these
chapters address the second research question.

_Chapter 6_ develops an evaluation framework for personalisation strategy
experiment designs. These experiment designs face unique challenges, such as
low test power and partial non-compliance from users, issues that we will
cover in Chapters 3 and 5. The evaluation framework enables one to compare two
experiment setups given the circumstances they face with their target
audience, thus addressing the third research question. We also derive
interpretable rules of thumb from the framework to enable experimenters to
compare typical experiment setups quickly.

We conclude in _Chapter 7_ with an outline of promising investigation pathways
arising from each workstream. In addition, we make all the code used for
experiments and simulations described in the thesis, as well as the ASOS
Digital Experiment Dataset, publicly available. Readers can find the links to
the code repositories and the dataset at the bottom of this page333The link to
code repositories and the dataset are as follows:  • Code repository
associated with the ranking under lower uncertainty framework (as described in
Chapter 2): https://github.com/liuchbryan/ranking_under_lower_uncertainty; •
ASOS Digital Experiments Dataset and accompanying datasheet (as described in
Chapter 4): https://osf.io/64jsb/. The OSF project also embeds a code
repository with experiments on the dataset (https://github.com/liuchbryan/oce-
dataset/); • Code repository and results on the two publicly available
datasets associated with e-commerce experiments with dependent responses (as
described in Section 5.5): https://github.com/liuchbryan/oce-ecomm-abv-
calculation; and • Code repository associated with the evaluation framework
for personalisation strategy experiment designs (as described in Chapter 6):
https://github.com/liuchbryan/experiment_design_evaluation. and again in the
footnotes of the relevant chapters.

###  1.4 Publications

This thesis contains research that has been published during PhD study. We
list where readers can find them in the thesis, together with details of the
associated conference and journal papers, below:

  * •

Chapter 2 and Appendix A.1: C. H. B. Liu and B. P. Chamberlain (2019). What is
the Value of Experimentation & Measurement? In: 2019 IEEE International
Conference on Data Mining (ICDM ’19), 1222–1227. DOI: 10.1109/ICDM.2019.00151.

  * •

Chapter 2 and Appendix A.1: C. H. B. Liu, B. P. Chamberlain and E. J. McCoy
(2020). What is the Value of Experimentation and Measurement? Data Science and
Engineering 5, 152–167. DOI: 10.1007/s41019-020-00121-5. Part of Special
Issue: Highly-Rated Short Papers of ICDM 2019.

  * •

Chapter 4: C. H. B. Liu, Â. Cardoso, P. Couturier and E. J. McCoy (2021).
Datasets for Online Controlled Experiments. In: Proceedings of the Neural
Information Processing Systems Track on Datasets and Benchmarks (NeurIPS
Datasets and Benchmarks ’21). URL:https://datasets-benchmarks-
proceedings.neurips.cc/paper/2021/file/274ad4786c3abca69fa097b85867d9a4-Paper-
round2.pdf.

  * •

Chapter 5 (Section 5.5): C. H. B. Liu and E. J. McCoy (2023). Measuring
e-Commerce Metric Changes in Online Experiments. In: Companion Proceedings of
the ACM Web Conference 2023 (WWW ’23 Companion). DOI: 10.1145/3543873.3584654.

  * •

Chapter 6, Appendix A.2, and Appendix A.3: C. H. B. Liu and E. J. McCoy
(2020). An Evaluation Framework for Personalization Strategy Experiment
Designs. arXiv: 2007.11638 [stat.ME]. Presented in AdKDD ’20 Workshop (in
conjunction with KDD ’20),awarded Best Student Paper.

###  1.5 List of Acronyms and Mathematical Symbols

Readers may encounter the following acronyms recurring in multiple thesis
sections. We will define the acronyms in the main text at least once before
referring to the acronyms.

  * •

ATE: Average treatment effect

  * •

CATE: Conditional average treatment effect

  * •

CI: Confidence interval

  * •

CDF: Cumulative density function

  * •

CLT: Central limit Theorem

  * •

CVR: Conversion rate

  * •

DEM: Digital experimentation and measurement (usually followed by the word
“capability/capabilities”)

  * •

MDE: Minimum detectable effect

  * •

NHST: Null hypothesis significance test(ing)

  * •

OCE: Online controlled experiment

  * •

PDF: Probability density function

  * •

RCT: Randomised controlled trial

We also list the mathematical symbols and the quantities/concepts they
represent below. Each entry is indexed by the leading symbol and follows the
following format:

(Applicable chapter(s)) _Symbol(s)_ – (Type) Quantity/concept the symbol
represents.

##### Applicable chapter(s)

This states the thesis chapter(s) that a notation applies to. The notations
used in Chapter 2 also apply to Appendix A.1, and those used in Chapter 6 also
apply to Appendices A.2 and A.3.

##### Symbol(s)

The thesis covers topics traditionally belonging to multiple statistical
disciplines. Thus, it is inevitable to see some notation clash. We follow the
order set out below when deciding which set of notations takes precedence:

  1. 1.

Elementary statistical constructs found in introductory statistical texts
(e.g., $X$ for a random variable, $f_{X}(\cdot)$ for the probability density
function of $X$, and $\mathbb{E}(X)=\mu_{X}$ for the expected value of $X$);

  2. 2.

Notation specific to statistical testing (e.g., $H_{0}$ and $H_{1}$ for null
and alternate hypotheses, $\alpha$ for the significance level, $\beta$ for the
power);

  3. 3.

Common notation in other sub-fields, including causal inference, order
statistics, and econometrics; and

  4. 4.

Specific notation introduced by individual research articles.

Notations with lower precedence are generally assigned a new symbol to
minimise confusion. That said, in a couple of cases, the same symbol may
represent two different quantities/concepts in the same chapter – their
meaning should be apparent given the context surrounding its use.

##### Type

We also abbreviate the types of quantities and concepts in the tables as
follows:

  * •

calc.: calculated quantity,

  * •

const.: constant,

  * •

dist.: distribution,

  * •

func.: function,

  * •

id.: identifier, and

  * •

r.v.: random variable.

Table 1.1: List of mathematical symbols and the quantity/concept represented by them (seven pages). See Section 1.5 for further information. | Leading
---
Symbol
Quantity/Concept Represented
$A/a$ |  |  (Chapter 5) $\mathcal{A}$ – (r.v.) Treatment Assignment variable in the potential outcomes framework, with realised values denoted $a$.
---
(Chapter 6) $A$ (and $A1$, $A2$) – (id.) An Analysis group in a
personalisation strategy experiment, often referenced together with $B$.
$B/b$ |  |  (Chapter 3) $B$ – (calc. / r.v.) Binomial test of proportion statistic.
---
(Chapters 3–4) $BF_{n,m}$ – (calc.) Bayes Factor (in a Bayesian test, upon
observing first $n$ and $m$ items in the first and second sample,
respectively).
(Chapter 6) $B$ (and $B1$, $B2$) – (id.) An analysis group in a
personalisation strategy experiment, often referenced together with $A$.
$C/c$ |  |  (All chapters) $\textrm{Cov}(\cdot\,,\cdot)$ – (func.) Covariance.
---
(Chapter 2) $c$ – (const.) Adjustment to the quantile of an order statistic.
(Chapter 2) $c$ – (const.) Risk-free value in the Sharpe ratio.
(Chapter 3) $C^{2}$ – (calc. / r.v.) $\chi^{2}$ goodness-of-fit test
statistic.
(Chapter 5) $\mathcal{C}$ – (r.v.) Covariate(s) under the potential outcomes
framework.
(Chapter 6) $C0,C1,C2,C3$ – (id.) User groups in a personalisation strategy
experiment that acts as Control.
(Appendix A.1) $C$ – (r.v.) Count.
$D/d$ |  |  (Chapter 2) $\mathcal{D}$ – (r.v.) Difference in value of items selected in the ranking under lower uncertainty problem.
---
(Chapters 3–4) $d$ – (calc.) Cohen’s $d$.
$E/e$ |  |  (All chapters) $\mathbb{E}(\cdot)$ – (func.) Expected value.
---
(Chapter 2) $\mathcal{E}_{i}$ / $\mathcal{E}_{(r)}$ – (r.v.) Estimated value
of item $i$ / $r$th-ranked item under estimation uncertainty in the ranking
under lower uncertainty problem, often referenced together with
$\mathcal{V}_{i}$ / $\mathcal{V}_{\mathcal{I}(r)}$.
(Chapters 3) $E_{i}$ – (r.v.) Expected frequency for category $i$ in a
$\chi^{2}$ goodness-of-fit test.
(Chapters 3–4) $E_{n,m}$ – (calc.) Effective sample size (in a Bayesian test,
upon observing first $n$ and $m$ items in the first and second sample,
respectively).

| Leading
---
Symbol
Quantity/Concept Represented
$F/f$ |  |  (All chapters) $F_{X}(\cdot)$ – (func.) Cumulative density function (CDF) of an r.v. $X$.
---
(All chapters) $f_{X}(\cdot)$ – (func.) Probability density function (PDF) of
an r.v. $X$.
(All chapters) $f^{\prime}_{X}(\cdot)$ – (func.) Derivative of the PDF of an
r.v. $X$.
(All chapters) $f(\cdot\,|\,\cdot)$ – (func.) Conditional PDF.
$G/g$ |  |  (Chapter 5) $g(\cdot)$ – (func.) A generic function that maps the treatment and covariate(s) to the observed/potential response under the potential outcomes framework.
---
(Chapter 6) $G$ – (id.) A generic user Group-scenario combination in a
personalisation strategy experiment. See Table 6.1 for possible values.
$H/h$ |  |  (Chapter 2) $\mathcal{H}_{i}$ / $\mathcal{H}_{(r)}$ – (r.v.) Estimated value of item $i$ / $r$th-ranked item under high estimation noise in the ranking under lower uncertainty problem, often referenced together with $\mathcal{V}_{i}$ / $\mathcal{V}_{\mathcal{I}(r)}$ and $\mathcal{L}_{j}$ / $\mathcal{L}_{(s)}$.
---
(Chapters 3–6) $H_{0}$, $H_{1}$ – (id.) Null and alternate hypotheses.
(Chapters 3–4) $H$ – (id.) The mixing distribution in a mixed Sequential
Probability Ratio Test (mSPRT).
(Chapters 3–4) $H$ – (id.) A generic statistical hypothesis in a Bayesian
test.
$I/i$ |  |  (All chapters) $i$ – (id.) Index identifying an r.v. within a set of i.i.d. r.v.’s, often referenced together with $j$.
---
(Chapter 2) $\mathcal{I}(\cdot)$ – (func.) Index function that maps the rank
of an r.v. within a set of independently identically distributed (i.i.d.)
r.v.’s $\mathcal{E}$ / $\mathcal{H}$ to its index, often referenced together
with $\mathcal{J}(\cdot)$.
(Chapter 6) $I1,I2,I\phi,I\psi$ – (id.) User groups in a personalisation
strategy experiment under Intervention.
(Chapter A.1) $\mathbb{I}$ – (r.v.) Indicator variable, often with the
condition expressed as a subscript.
| Leading
---
Symbol
Quantity/Concept Represented
$J/j$ |  |  (All chapters) $j$ – (id.) Index identifying an r.v. within a set of i.i.d. r.v.’s, often referenced together with $i$.
---
(Chapter 2) $\mathcal{J}(\cdot)$ – (func.) Index function that maps the rank
of an r.v. within a set of independently identically distributed (i.i.d.)
r.v.’s. $\mathcal{L}$ to its index, often referenced together with
$\mathcal{I}(\cdot)$.
$L/l$ |  |  (Chapter 2) $\mathcal{L}_{i}$ / $\mathcal{L}_{(s)}$ – (r.v.) Estimated value of item $i$ / $s$th-ranked item under low estimation noise in the ranking under lower uncertainty problem, often referenced together with $\mathcal{V}_{i}$ / $\mathcal{V}_{\mathcal{J}(s)}$ and $\mathcal{H}_{i}$ / $\mathcal{H}_{(r)}$.
---
(Appendix A.1) $\mathcal{L}^{*}$ – (r.v.) $\mathcal{L}_{\mathcal{I}(r)}$
normalised by the mean and variance of $\mathcal{L}_{j}$ under normal
assumptions.
$M/m$ |  |  (Chapter 2) $M$ – (const.) Number of items one can select (i.e., capacity) in the ranking under lower uncertainty problem.
---
(Chapters 3–6) $m$ – (const.) Number of items in (or size of) a statistical
sample $Y$, either total or that observed so far within a sequential/Bayesian
test setting; often referenced together with $n$.
$N/n$ |  |  (All chapters) $\mathcal{N}(\cdot\,,\cdot)$ – (dist.) A normal distribution, with its mean and variance as parameters.
---
(Chapter 2) $N$ – (const.) Number of items available for selection in the
ranking under lower uncertainty problem.
(Chapters 3–6) $n$ – (const.) Number of items in (or size of) a statistical
sample $X$, either total or that observed so far within a sequential/Bayesian
test setting; often referenced together with $m$.
(Chapter 6) $n_{\mathcal{G}}$ – (const.) Number of samples in a user/analysis
group $\mathcal{G}$, where $\mathcal{G}\in\\{0,1,2,3,A,B,A1,A2,B1,B2\\}$, in a
personalisation strategy experiment.
$O/o$ |  |  (Chapters 3) $O_{i}$ – (r.v.) Observed frequency for category $i$ in a $\chi^{2}$ goodness-of-fit test.
---
(Chapter 6) $O(\cdot)$ – (func.) Big-O notation describing a function’s
limiting behaviour.
$P/p$ |  |  (Chapter 2) $\mathbb{P}(\cdot)$ – (func.) Probability.
---
(Chapters 3–6) $p$ – (calc.) $p$-value of a null hypothesis significance test.
| Leading
---
Symbol
Quantity/Concept Represented
$R/r$ |  |  (Chapter 2) $r$ – (id.) Rank of an r.v. (in order statistics terms) within a set of r.v.’s $\mathcal{E}$ / $\mathcal{H}$, often referenced together with $s$.
---
(Chapter 6) $R$ – (id.) A personalisation strategy experiment setup, often
referenced together with $S$.
$S/s$ |  |  (Chapter 2) $s$ – (id.) Rank of an r.v. (in order statistics terms) within a set of r.v.’s $\mathcal{L}$, often referenced together with $r$.
---
(Chapters 3) $S_{n}$ – (r.v.) Sequential probability ratio test (SPRT)
statistic.
(Chapters 3–4) $S(\cdot\,,\cdot)$ – (func.) Sign function (in a Mann-Whitney
$U$ test).
(Chapters 3–6) $s^{2}_{X}$ – (calc.) Sample variance of a sample $X$.
(Chapter 6) $S$ – (id.) A personalisation strategy experiment Setup, often
referenced together with $R$.
$T/t$ |  |  (All chapters) $t_{\nu}$ – (dist.) A Student’s $t$ distribution with $\nu$ degrees of freedom.
---
(All chapters) $T_{\nu}(\cdot)$ – (func.) CDF of a Student’s $t$-distribution
with $\nu$ d.f.
(All chapters) $t_{\nu,q}$ – (calc.) $q$th quantile of a Student’s
$t$-distribution with $\nu$ d.f.
(Chapter 2) $T_{i}$ / $T_{i}^{\prime}$ – (r.v.) Student’s $t$-distributed
r.v.’s.
(Chapters 3–6) $T$ – (calc. / r.v.) Test statistic (generic, or that of a
$t$-test).
(Chapter 4) $t,t+1,...$ – (id.) Points in time (in statistical tests with
adaptive stopping).
(Appendix A.1) $T(\cdot\,,\cdot)$ – (func.) Owen’s $T$-function.
$U/u$ |  |  (Chapters 2 & 6) $U(a,b)$ – (dist.) A uniform distribution with bounds $a$ and $b$.
---
(Chapters 3–4) $U$ – (calc. / r.v.) Mann-Whitney $U$-test statistic.
$V/v$ |  |  (All chapters) $\textrm{Var}(\cdot)$ – (func.) Variance.
---
(Chapter 2) $\mathcal{V}_{i}$, $\mathcal{V}_{\mathcal{I}(r)}$,
$\mathcal{V}_{\mathcal{J}(s)}$ – (r.v.) True Value of items $i$,
$\mathcal{I}(r)$, and $\mathcal{J}(s)$ in the ranking under lower uncertainty
problem, often referenced together with $\mathcal{E}_{i}$ /
$\mathcal{E}_{(r)}$, $\mathcal{H}_{i}$ / $\mathcal{H}_{(r)}$, and
$\mathcal{L}_{j}$ / $\mathcal{L}_{(s)}$.
(Chapters 3–4) $V^{2}$ – (const.) Hyperparameter, in a Bayesian test, which
represents the variance of the effect size prior.
| Leading
---
Symbol
Quantity/Concept Represented
$W/w$ |  |  (Chapter 2) $\mathcal{W}$, $\mathcal{W}_{1}$, $\mathcal{W}_{2}$ – (r.v.) Average true value of items selected under some estimation uncertainty, high estimation noise, and low estimation noise, respectively, in the ranking under lower uncertainty problem.
---
(Chapters 3–4) $W_{n,m}$ – (calc. / r.v.) Wald test statistic (in a
sequential/Bayesian test, upon observing first $n$ and $m$ items in the first
and second sample, respectively.
$X/x$ |  |  (Chapters 3–6) $X_{i}$ – (r.v.) Item $i$ in a sample $X$, often referenced together with $Y_{j}$ in two-sample testing.
---
(Chapters 3–6) $\bar{X}$ – (r.v.) Sample mean of a sample $X$.
(Chapters 3–6) $\bar{X}_{n}$ – (r.v.) Sample mean of a sample $X$ (in a
sequential/Bayesian test, upon observing the first $n$ items in the sample).
(Chapters 3–6) $\bar{x}$ – (calc.) Realised sample mean of a sample $X$.
(Chapters 5) $X(a)$ / $X_{i}(a)$ – (r.v.) Potential response/outcome of
$X$/$X_{i}$ under the treatment assignment $a\in\\{0,1,...\\}$ in the
potential outcomes framework, often referenced together with $Y(a)$ /
$Y_{j}(a)$.
$Y/y$ |  |  (Chapters 3–6) $Y_{j}$ – (r.v.) Item $j$ in a sample $Y$, often referenced together with $X_{i}$ in two-sample testing.
---
(Chapters 3–6) $\bar{Y}$ – (r.v.) Sample mean of a sample $Y$.
(Chapters 3–6) $\bar{Y}_{n}$ – (r.v.) Sample mean of a sample $Y$ (in a
sequential/
Bayesian test, upon observing the first $n$ items in the sample).
(Chapters 5) $Y(a)$ / $Y_{j}(a)$ – (r.v.) Potential response/outcome of
$Y$/$Y_{j}$ under the treatment assignment $a\in\\{0,1,...\\}$ in the
potential outcomes framework, often referenced together with $X(a)$ /
$X_{i}(a)$.
$Z/z$ |  |  (Chapters 3–6) $z_{q}$ – (calc.) $q$th quantile of a standard normal r.v.
---
(Chapter 3) $Z$ – (calc. / r.v.) Standard score / $z$-test statistic.
(Chapter 6) $z$ – (id.) Mathematical shorthand – the difference between two
normal quantiles (see Equation (6.14)).
(Appendix A.1) $Z$ – (r.v.) A standard normal r.v.
| Leading
---
Symbol
Quantity/Concept Represented
$\alpha$ |  |  (Chapters 3–6) $\alpha$ – (const.) Significance level of a statistical test.
---
(Appendix A.1) $\alpha_{\mathcal{P}}$ – (const.) Beta distribution parameter,
often referenced together with $\beta_{\mathcal{P}}$.
$\beta$ |  |  (Chapters 3–6) $\beta_{\theta}$ – (const.) Power of a statistical test under a specific hypothesis with parameter $\theta$.
---
(Appendix A.1) $\beta_{\mathcal{P}}$ – (const.) Beta distribution parameter,
often referenced together with $\alpha_{\mathcal{P}}$.
$\Delta/\delta$ |  |  (All chapters) $\Delta$ – (calc. / r.v.) Effect size.
---
(Chapters 3–4) $\Delta^{\\!\circ}$ – (calc. / r.v.) Effect size standardised
by the pooled variance in a Bayesian test.
(Chapters 3–4) $\delta_{n,m}$ – (calc. / r.v.) “Test statistic” of a Bayesian
test, upon observing the first $n$ and $m$ items in the first and second
sample, respectively.
(Chapter 6) $\Delta_{S}$ – (calc.) Actual effect size of a personalisation
strategy setup $S$.
$\epsilon$ |  |  (Chapter 2) – $\epsilon,\epsilon_{1},\epsilon_{2}$ – (r.v.) Estimation noise.
---
$\eta$ |  |  (Chapter 6) $\eta$ – (id.) Mathematical shorthand (see Equation (6.14)).
---
$\Theta/\theta$ |  |  (All chapters) $\theta$ – (const.) Parameter of interest in a hypothesis test.
---
(All chapters) $\theta_{0}$ – (const.) Value of the parameter of interest
under the null hypothesis.
(All chapters) $\theta^{*}$ – (const.) A specific parameter of interest.
(Chapter 6) $\theta^{*}_{S}$ – (const.) Minimum detectable effect size of a
personalisation strategy experiment setup $S$.
$\Lambda/\lambda$ |  |  (Chapters 3–4) $\Lambda^{H,\theta_{0}}_{n}$ / $\tilde{\Lambda}^{H,\theta_{0}}_{n}$ – (calc. / r.v.) One-sample / two-sample mixed sequential probability ratio test (mSPRT) statistic with mixing distribution $H$ and null hypothesis parameter $\theta_{0}$.
---
$\mu$ |  |  (All chapters) $\mu_{X}$ – (const.) Mean of an r.v., a population, or a group $X$.
---
$\nu$ |  |  (All chapters) $\nu$ – (const.) Degrees of freedom (d.f.).
---
$\Xi/\xi$ |  |  (Chapter 6) $\xi$ – (id.) Mathematical shorthand (see Equation (6.14)).
---
| Leading
---
Symbol
Quantity/Concept Represented
$\Pi/\pi$ |  |  (Chapter 3) $\pi_{X}$ – (const.) Success rate of a Bernoulli distributed r.v. $X$.
---
(Chapter 6) $\pi_{\textrm{min}}$ – (const.) Minimum test power.
$\rho$ |  |  (All chapters) $\rho_{XY}$ – (func.) Correlation between two r.v.’s $X$ and $Y$.
---
$\Sigma/\sigma$ |  |  (All chapters) $\sigma_{X}$ – (const.) Standard deviation of an r.v., a population, or a group $X$.
---
(All chapters) $\sigma^{2}_{X}$ – (const.) Variance of an r.v., a population,
or a group $X$.
$\tau$ |  |  (Chapters 3–4) $\tau^{2}$ – (const.) Hyperparameter in a mixed sequential probability ratio test (mSPRT), representing the variance of the mixture distribution.
---
$\Phi/\phi$ |  |  (All chapters) $\Phi(\cdot)$ – (func.) Cumulative density function (CDF) of a standard normal r.v.
---
(All chapters) $\phi(\cdot)$ / $\phi(\,\cdot\,;\mu,\sigma^{2})$ – (func.)
Probability density function (PDF) of a standard normal r.v. / a normal r.v.
with mean $\mu$ and variance $\sigma^{2}$.
$\chi$ |  |  (Chapter 3) $\chi^{2}_{\nu}$ – (dist.) A $\chi^{2}$-distribution with $\nu$ d.f.
---

##  Chapter 2 What is the Value of Digital Experimentation and Measurement
Capabilities?

From the thesis title, it is reasonable for readers to expect content covering
methodological advances in statistical tests, experiment design, and causal
inference. While we will address such mainstream topics in later chapters of
the thesis, we first present a ranking under lower uncertainty problem as an
application of order statistics to motivate our topics. As the chapter title
suggests, addressing the problem enables us to value digital experimentation
and measurement (DEM) capabilities quantitatively, which, in turn, helps us to
answer why we require such capabilities in the first place.

This chapter is adapted from the research paper “What is the Value of
Experimentation & Measurement?” It was first presented at the 2019 IEEE
International Conference on Data Mining (ICDM) [174] and later extended and
published in Data Science and Engineering as part of a special issue on
Highly-Rated Short Papers of ICDM 2019 [175]. In Sections 2.1 and 2.2, as well
as introducing related work, we will motivate how the requirement to value DEM
capabilities arises and leads to our upcoming ranking under lower uncertainty
problem. Readers who are more interested in the mathematical formulation of
the ranking under lower uncertainty problem can skip straight to Section 2.3,
though they may find the chapter organisation at the end of Section 2.1
useful.

###  2.1 Motivation

The value of DEM capabilities is currently best reflected in the success of
organisations that have adopted and advocated for them in the past decade.
Many major technology companies report having mature infrastructure for online
controlled experiments (OCEs, e.g., Google [260], Linkedin [288], and
Microsoft [149]) or are heavily investing in state-of-the-art techniques
(e.g., Airbnb [161], Netflix [286], and Yandex [222]), or both. Amazon [123],
Facebook [105], and Uber [20] have also reported the use of various causal
inference techniques to measure the incrementality of advertising campaigns.
Several start-ups (e.g. Optimizely [134] and Qubit [30]) have also recently
been established purely to manage OCEs for businesses.

While mature DEM capabilities can quantify the value of a business
proposition, it remains a substantial challenge to “measure the measurer” – to
quantify the value of the DEM capabilities themselves. To the best of our
knowledge, no work addresses the question, “Should we invest in DEM
capabilities?” or how to value these capabilities when we first present our
ideas. The lack of prior work makes it hard to build a compelling business
case to justify investment in the related personnel and infrastructure. We
address this problem by calculating both the expected value and the risk,
allowing the Sharpe ratio [246] for a DEM capability to be calculated and
compared to other potential investments.

(a) High estimation noise

(b) Low estimation noise

Figure 2.1: Prioritising four projects (the fruits) according to their value
(x-axes). The semi-opaque icons represent the projects’ true value, and the
solid icons represent possible project value estimates under some level of
uncertainty (horizontal lines) in the estimation process. (Top) Under a noisy
estimation process, projects with a low true value (e.g., project apple) may
appear to have a high value and be prioritised erroneously. (Bottom) DEM
reduces the estimation noise, enabling a better prioritisation with value
estimates closer to the truth. The figures include icons designed by
Smashicons from Flaticon.com.

The value created by DEM capabilities comes from the following three sources:

  1. 1.

Recognising value: DEM capabilities enable one to attribute value to a digital
product, business proposition or service. They also prevent damage from
business propositions that have a negative value. This is important for
dynamic organisations with large numbers of business propositions, as the
damage caused by individual rollouts can be compartmentalised and contained in
a similar fashion, similar to unit and integration testing in software
development.

  2. 2.

Prioritisation: Without DEM capabilities, one prioritises based on back-of-
envelope estimates or gut feel, which has high uncertainty. DEM reduces the
magnitude of the noise arising from estimation, enabling prioritisation based
on estimates closer to the true values and improving long-term decision-making
(see Figure 2.1).

  3. 3.

Optimisation: DEM capabilities allow one to evaluate large numbers of variants
against each other and the best to be selected efficiently. Without such
capabilities, we can still experiment with different business propositions
sequentially, though it is slow and introduces noise from the changing
environment.

Quantifying the value of a DEM capability is relatively straightforward once
it is in place. For example, the value of Item 1 comes from rolling back
negative business propositions: we can calculate it by summing the negative
contributions of unsuccessful business propositions. Likewise, we can
calculate the value of Item 3 by summing the difference between the maximum
and the mean value for each variant over the business propositions. Indeed,
these are the approaches taken by [182] when valuing LinkedIn’s
experimentation platform using past experiment data.

The approaches above are not feasible for organisations building DEM
capabilities from scratch. One can attempt to estimate the values from Items 1
and 3 using generic value distribution across business propositions, which is
given across industries in [136] and [30], though these estimates are
generally no better than back-of-envelope calculations. Without an accurate
valuation, organisations remain in a chicken and egg situation – no valuation,
no investment; no investment, no capability; no capability, no valuation.

Given the above, we see quantifying the value of Item 2 as the less explored
yet more compelling approach and the subject of the remainder of this chapter.
DEM capabilities improve prioritisation by reducing uncertainty in the value
estimates of each business proposition. This is a form of ranking under
uncertainty, a well-studied problem in statistics and operations research.
However, in all previous work, the variance is assumed to be a fixed constant
or changed without measuring the corresponding change in the value of the
ranked items. Here, we wish to understand the value of ranking under _lower_
uncertainty through DEM capabilities.

Our contribution is as follows. We

  1. 1.

Specify the first model that values the contribution of a DEM capability in
terms of better prioritisation due to reduced estimation noise for business
propositions (Sections 2.3 and 2.4);

  2. 2.

Derive the variance of our estimator, allowing one to calculate a Sharpe ratio
to guide organisations considering an investment in DEM (Section 2.5); and
finally

  3. 3.

Provide two case studies based on large-scale meta-analyses that reflect how
one can apply our model to real-world practice (Section 2.7) and two
extensions that open the door to future work in this area (Section 2.8).

###  2.2 Related work

There is a vast literature on using controlled or natural experiments in a
digital technology context. In addition to that mentioned in Section 2.1,
there are works dedicated to running trustworthy online controlled experiments
[74], choosing good metrics [126] and designing experiments where samples are
dependent due to external confounders [18, 19].111We will describe the works
in greater detail in Chapter 5. However, these works all assume the existence
of DEM capabilities, and to the best of our knowledge, no literature that
helps organisations justify the acquisition of DEM capabilities exists. While
[182] asked a similar question as this chapter does, they aimed to value an
experimentation platform, i.e., an existing DEM capability. We believe that
filling this gap is necessary for widespread adoption and that increased
participation will accelerate the development of the field.

This chapter is related to existing work in statistics and operations
research, particularly on decision-making under uncertainty, which has been
extensively studied since the 1980s. Notable work includes proposals for
additional components in a decision maker’s utility function [22], alternate
risk measures [289], and a general framework for decision-making with
incomplete information (i.e., uncertainty) [278]. These works assume the
inability to change the noise associated with estimation or measurement (or
both).

The sub-problem of ranking under uncertainty has also attracted considerable
attention, partially due to the advent of large databases and the requirement
to rank results with a certain ambiguity in relevance [255]. While [298]
measured the influence of noise levels in their work, they focused on the
quality of the ranks themselves but not the value associated with the ranks.

The project selection problem is a related topic in optimisation, where the
goal is to find the optimal set of business propositions using mixed integer
linear programming, possibly under uncertainty. Work in this domain generally
seeks methods that cope with existing risk/noise [186], and to the best of our
knowledge, no work considers the value of reducing risk. While [245] have
discussed lowering the uncertainty level during the selection process, they
refer to the uncertainty of decision parameters instead of the general noise
level.

###  2.3 Mathematical Formulation

We formulate the ranking under lower uncertainty problem, which has a wide
variety of important applications in its own right [251, 298], as follows.
Consider the scenario where we select $M$ business propositions from $N$
candidates, where $M<N$. The Estimated value of each business proposition is
given by ${\mathcal{E}_{i}=\mathcal{V}_{i}+\epsilon_{i}}$, where
$\mathcal{V}_{i}$ are the true yet unobserved Values estimated with error
$\epsilon_{i}$. The business propositions are labelled in ascending order of
estimated value $\mathcal{E}_{i}$ to get the order statistics
${\mathcal{E}_{(1)},\mathcal{E}_{(2)},...,\mathcal{E}_{(N)}}$, and we select
the $M$ business proposition with the highest estimated values:
${\mathcal{E}_{(N-M+1)},\mathcal{E}_{(N-M+2)},...,\mathcal{E}_{(N)}}$. We are
interested in the true value of the selected business propositions, given by

$\displaystyle\mathcal{V}_{\mathcal{I}(N-M+1)},\mathcal{V}_{\mathcal{I}(N-M+2)},...,\mathcal{V}_{\mathcal{I}(N)}\,,$
(2.1)

where $\mathcal{I}(\cdot)$ denotes the index function that maps the ranking to
the index of the business proposition. Note one should not confuse the set in
Expression (2.1) with the set $\\{\mathcal{V}_{(N-M+1)},$
$\mathcal{V}_{(N-M+2)},...,\mathcal{V}_{(N)}\\}$ – the latter denotes the top
$M$ business propositions ranked by their true value and are likely to be
different [298].

We define the mean true value of the $M$ selected business propositions as

$\displaystyle\mathcal{W}\triangleq\frac{1}{M}\left(\mathcal{V}_{\mathcal{I}(N-M+1)}+\mathcal{V}_{\mathcal{I}(N-M+2)}+...+\mathcal{V}_{\mathcal{I}(N)}\right)\,,$
(2.2)

where the best prioritisation maximises $\mathcal{W}$.222Readers can
internalise the $\mathcal{W}$ notation by drawing parallels between the name
of the letter in French (double vé) and Spanish (doble ve) – both literally
meaning “double v” – and the fact that we use $\mathcal{W}$ to denote an
aggregation of two or more $\mathcal{V}$.

Part of the value of ranking under lower uncertainty, which DEM capabilities
help bring, arises from the observation that $\mathcal{W}$ increases when the
magnitude of the uncertainties arising from estimation ($\epsilon_{i}$)
decreases. We are interested in the value gained by reducing estimation
uncertainty without changing the set of business propositions (i.e., retaining
all $\mathcal{V}_{i}$), as the true value of the business propositions does
not depend on the measurement method used:

$\displaystyle\mathcal{D}\triangleq\mathcal{W}\,|_{\textrm{ lower
noise}}-\mathcal{W}\,|_{\textrm{ higher noise}}\,.$ (2.3)

Obtaining the value gained ($\mathcal{D}$) enables us to compare DEM
capabilities with other potential investments. The Sharpe ratio is a standard
quantity used in finance to compare different investments. This standard
score-like ratio represents the risk-adjusted expected excess return on
investment compared to a risk-free investment [246]. In our case,
$\mathbb{E}(\mathcal{D})$ represents the expected return on investment of DEM
capabilities, $\textrm{Var}(\mathcal{D})$ represents the risk, and we can
specify the return on investment of the risk-free investment as a constant
$c$. This enables us to calculate the Sharpe ratio as

$\displaystyle\frac{\mathbb{E}(\mathcal{D})-c}{\sqrt{\textrm{Var}(\mathcal{D})}}\,.$
(2.4)

####  2.3.1 Modelling values with statistical distributions

To value a DEM capability using such a generic framework that one can apply in
many different ways across diverse organisations, it is first necessary to
make some simplifying assumptions about the statistical properties of the
business propositions under consideration. We assume the value of the business
propositions ($\mathcal{V}_{i}$) and the estimation noises ($\epsilon_{i}$)
are randomly distributed:

$\displaystyle\mathcal{V}_{i}\overset{\textrm{i.i.d.}}{\sim}F_{\mathcal{V}}(\cdot)\,,\textrm{
where }$
$\displaystyle\mathbb{E}(\mathcal{V}_{i})=\mu_{\mathcal{V}},\textrm{Var}(\mathcal{V}_{i})=\sigma^{2}_{\mathcal{V}}\,,$
$\displaystyle\epsilon_{i}\overset{\textrm{i.i.d.}}{\sim}F_{\epsilon}(\cdot)\,,\textrm{
where }$
$\displaystyle\mathbb{E}(\epsilon_{i})=\mu_{\epsilon},\textrm{Var}(\epsilon_{i})=\sigma^{2}_{\epsilon}\,,$
(2.5)

where $\mathcal{V}_{i}\perp\epsilon_{j}\,\forall\,i,j$ (see Figure 2.2).

Figure 2.2: The generative model in the ranking under lower uncertainty
problem in plate notation. $\mathcal{V}_{i}$ represents the true, unobserved
values of the items to be ranked. $\mathcal{E}_{i}$ represents the observed
values under some estimation noise level $\sigma^{2}_{\epsilon}$.

We note two special cases, one when both the value and the noises are assumed
to be normally distributed:

$\displaystyle\mathcal{V}_{i}\overset{\textrm{i.i.d.}}{\sim}\mathcal{N}(\mu_{\mathcal{V}},\sigma^{2}_{\mathcal{V}})\,,\;\epsilon_{i}\overset{\textrm{i.i.d.}}{\sim}\mathcal{N}(\mu_{\epsilon},\sigma^{2}_{\epsilon})\,;$
(2.6)

and the other when both the value and the noises are assumed to follow some
Generalized Student’s $t$-distributions:

$\displaystyle\mathcal{V}_{i}=\mu_{\mathcal{V}}+\sigma_{\mathcal{V}}\sqrt{\frac{(\nu-2)}{\nu}}T_{i}\,,\quad
T_{i}\overset{\textrm{i.i.d.}}{\sim}t_{\nu}\,,$
$\displaystyle\epsilon_{i}=\mu_{\epsilon}+\sigma_{\epsilon}\sqrt{\frac{(\nu-2)}{\nu}}T^{{}^{\prime}}_{i}\,,\quad
T^{{}^{\prime}}_{i}\overset{\textrm{i.i.d.}}{\sim}t_{\nu}\,,$ (2.7)

where $t_{\nu}$ is a standard Student’s $t$-distribution with $\nu$ degrees of
freedom. The location and scaling parameters ensure $\mathcal{V}_{i}$ and
$\epsilon_{i}$ have the mean and variance specified in Expression (2.5).

These two cases are particularly relevant as meta-analyses compiled on the
results of 6,700 e-commerce [30] and 432 marketing experiments [136],
respectively, indicate the uplifts measured by the experiments, and hence the
value of the business propositions under some estimation noise, exhibit the
following properties:

  1. 1.

They can be positive or negative,

  2. 2.

They are usually clustered around an average instead of uniformly spreading
across a specific range, and

  3. 3.

The distributions are heavy-tailed.

The normal assumptions cover the first two properties only, yet enable one to
draw on the wealth of results in order statistics and Bayesian inference
related to normal distributions to get started. The $t$-distributed
assumptions also cover property 3, though valuation under such assumptions is
more complicated as $t$-distributions do not have conjugate priors.

We will include the valuation under $t$-distributed assumptions under the
general case for brevity. We will, however, present empirical results in
Section 2.8.1, showing that the value gained under $t$-distributed assumptions
has a higher mean and variance, thus demonstrating that the model can capture
the “higher risk, higher reward” concept.

####  2.3.2 Key results

In the following two sections, we will derive the expected value and variance
for $\mathcal{W}$, the mean true value of the top $M$ business propositions
selected after being ranked by their estimated value (as defined in Equation
(2.2)), as well as the expected value and the variance of $\mathcal{D}$, the
value gained when the estimation noise is reduced. This enables us to
calculate the Sharpe ratio defined in Expression (2.4).

We will also provide two key insights. Firstly, the expected mean true value
of the selected business propositions ($\mathcal{W}$) increases when the
estimation noise ($\sigma^{2}_{\epsilon}$) decreases, and the relative
increase in value depends on how much noise we can reduce. Secondly, when $M$
is small, reducing the estimation noise may not lead to a statistically
significant improvement in the true value of the business propositions
selected. As a result, improvements in prioritisation driven by DEM may only
be justified for larger organisations.

###  2.4 Calculating the Expectation

We first derive the expected value for $\mathcal{D}$. This requires the
expected values of, in order:

  1. 1.

$\mathcal{E}_{(r)}$ – the _estimated_ value of the $r^{\textrm{th}}$ ranked
business proposition in estimated value;

  2. 2.

$\mathcal{V}_{\mathcal{I}(r)}$ – the _true_ value of the $r^{\textrm{th}}$
ranked business proposition in estimated value; and

  3. 3.

$\mathcal{W}$ – the mean of the _true_ value for the $M$ most valuable
business propositions ranked by their estimated values.

To obtain the expected value for $\mathcal{E}_{(r)}$, we apply a result in
Section 4.6 of [56], which approximates the expected value of the order
statistics $\mathcal{E}_{(r)}$ using the quantile function of
$\mathcal{E}_{i}$:

$\displaystyle\mathbb{E}(\mathcal{E}_{(r)})\approx
F_{\mathcal{E}}^{-1}\Big{(}\frac{r-c}{N-2c+1}\Big{)},$ (2.8)

where $F_{\mathcal{E}}^{-1}(\cdot)$ denotes the quantile function for
$\mathcal{E}_{i}$ and $c$ is a constant correcting the quantile of the
ranks.333Many values for $c$ were proposed. Early works advocate $c=0$ or
$c=0.5$ depending on how one approaches continuity correction between ranks
and quantiles [119]. [27] proposes a compromise value of $c=\frac{3}{8}$ based
on a tabulation of $c$ required to yield the correct expected value for
different $r$ and $N$. [119] expands the tabulation and proposes using a
different $c$ for each $N$, with the values of $c$ hovering around 0.4. For
simplicity, we take $c=0.4$ for all $N$ in all our calculations. The formula
results from applying the first-order delta method on the underlying compound
beta-$F^{-1}_{\mathcal{E}}$ distribution [223]. Such estimation will
inevitably incur a small bias, particularly for extreme order statistics
(i.e., $r$ close to one or $N$) and when $N$ is small. However, as verified
empirically in Section 2.6, we consider such an estimation accurate enough for
our application.444[119] also noted for normal order statistics, the maximum
error is 0.018 (i.e., less than one percent relative to the value of extreme
order statistics) for some choice of $c$. One can also include higher-order
terms in the corresponding Taylor expansion to improve estimation accuracy,
though we believe the additional complexity and, thus, reduced
interpretability in the resultant formula is not worth the marginal gain in
accuracy.

We then obtain the expected value of $\mathcal{V}_{\mathcal{I}(r)}$ by
applying Equation 6.8.3a of [56]:

$\displaystyle\mathbb{E}(\mathcal{V}_{\mathcal{I}(r)})=\,$
$\displaystyle\mu_{\mathcal{V}}+\rho_{\mathcal{V}\mathcal{E}}\,\sigma_{\mathcal{V}}\,\mathbb{E}\left(\frac{\mathcal{E}_{(r)}-(\mu_{\mathcal{V}}+\mu_{\epsilon})}{\sqrt{\sigma^{2}_{\mathcal{V}}+\sigma^{2}_{\epsilon}}}\right)$
$\displaystyle=\,$
$\displaystyle\mu_{\mathcal{V}}+\frac{\sigma^{2}_{\mathcal{V}}}{\sigma^{2}_{\mathcal{V}}+\sigma^{2}_{\epsilon}}\left(\mathbb{E}(\mathcal{E}_{(r)})-(\mu_{\mathcal{V}}+\mu_{\epsilon})\right),$
(2.9)

where
$\rho_{\mathcal{V}\mathcal{E}}=\sqrt{\sigma^{2}_{\mathcal{V}}}/\sqrt{\sigma^{2}_{\mathcal{V}}+\sigma^{2}_{\epsilon}}$
is the correlation between $\mathcal{V}_{i}$ and $\mathcal{E}_{i}$.

Equation (2.9) shows that decreasing the estimation noise
$\sigma^{2}_{\epsilon}$ will increase
$\mathbb{E}(\mathcal{V}_{\mathcal{I}(r)})$ for any ${r>(N+1)\cdot
F_{\mathcal{E}}(\mu_{\mathcal{V}}+\mu_{\epsilon})}$. It follows that the mean
true value of the top $M$ business propositions, selected according to their
estimated value, will generally increase with a lower estimation noise. We
show this by applying the expectation function to $\mathcal{W}$ defined in
Equation (2.2) to obtain

$\displaystyle\mathbb{E}(\mathcal{W})=\,$
$\displaystyle\frac{1}{M}\sum_{r=N-M+1}^{N}\mathbb{E}(\mathcal{V}_{\mathcal{I}(r)})$
$\displaystyle=\,$
$\displaystyle\mu_{\mathcal{V}}+\frac{\sigma^{2}_{\mathcal{V}}}{\sigma^{2}_{\mathcal{V}}+\sigma^{2}_{\epsilon}}\left[\left(\frac{1}{M}\sum_{r=N-M+1}^{N}\mathbb{E}(\mathcal{E}_{(r)})\right)-\left(\mu_{\mathcal{V}}+\mu_{\epsilon}\right)\right]\,.$
(2.10)

We finally consider the improvement when we reduce the estimation noise from
$\sigma^{2}_{\epsilon}=\sigma^{2}_{1}$ to
$\sigma^{2}_{\epsilon}=\sigma^{2}_{2}$. This will be the expected value gained
by having better DEM capabilities:

$\displaystyle\mathbb{E}(\mathcal{D})=\,$
$\displaystyle\mathbb{E}\left(\mathcal{W}\,|\,{\sigma^{2}_{\epsilon}=\sigma^{2}_{2}}\right)-\mathbb{E}\left(\mathcal{W}\,|\,{\sigma^{2}_{\epsilon}=\sigma^{2}_{1}}\right)$
$\displaystyle=\,$
$\displaystyle\frac{\sigma^{2}_{\mathcal{V}}}{\sigma^{2}_{\mathcal{V}}+\sigma^{2}_{2}}\left[\left(\frac{1}{M}\sum_{r=N-M+1}^{N}\mathbb{E}\left(\mathcal{E}_{(r)}\,|\,{\sigma^{2}_{\epsilon}=\sigma^{2}_{2}}\right)\right)-(\mu_{\mathcal{V}}+\mu_{\epsilon})\right]-$
$\displaystyle\quad\frac{\sigma^{2}_{\mathcal{V}}}{\sigma^{2}_{\mathcal{V}}+\sigma^{2}_{1}}\left[\left(\frac{1}{M}\sum_{r=N-M+1}^{N}\mathbb{E}\left(\mathcal{E}_{(r)}\,|\,{\sigma^{2}_{\epsilon}=\sigma^{2}_{1}}\right)\right)-(\mu_{\mathcal{V}}+\mu_{\epsilon})\right]\,.$
(2.11)

####  2.4.1 Expectation under normal assumptions

In the special case where $\mathcal{E}_{i}$ are normally distributed (with
mean $\mu_{\mathcal{V}}+\mu_{\epsilon}$ and variance
$\sigma^{2}_{\mathcal{V}}+\sigma^{2}_{\epsilon}$), the expected value for the
normal order statistics $\mathcal{E}_{(r)}$ is approximately

$\displaystyle\mathbb{E}(\mathcal{E}_{(r)})\approx\mu_{\mathcal{V}}+\mu_{\epsilon}+\sqrt{\sigma^{2}_{\mathcal{V}}+\sigma^{2}_{\epsilon}}\;\Phi^{-1}\left(\frac{r-c}{N-2c+1}\right),$
(2.12)

where $\Phi^{-1}(\cdot)$ denotes the quantile function of a standard normal
distribution [27]. It is worth noting that decreasing the estimation noise
$\sigma^{2}_{\epsilon}$ will decrease $\mathbb{E}(\mathcal{E}_{(r)})$ for any
$r>\frac{N+1}{2}$, appearing to lower the average value of the top $M$
business propositions. This is a common pitfall – we are not optimising the
combined estimated value of the selected business propositions. What matters
is the true yet unobserved value of that proposition,
$\mathcal{V}_{\mathcal{I}(r)}$, as shown below.

For $\mathcal{V}_{\mathcal{I}(r)}$, we can simplify Equation (2.9) by
substituting Equation (2.12). We can also evaluate from first principles by
noting a standard result in Bayesian inference, which states that the
posterior distribution of $\mathcal{V}_{i}$ once $\mathcal{E}_{i}$ is observed
is also normally distributed with mean

$\displaystyle\mu_{\mathcal{V}_{i}\,|\,\left(\mathcal{E}_{i}=e\right)}$
$\displaystyle=\frac{\sigma^{2}_{\mathcal{V}}}{\sigma^{2}_{\mathcal{V}}+\sigma^{2}_{\epsilon}}\,(e-\mu_{\epsilon})+\frac{\sigma^{2}_{\epsilon}}{\sigma^{2}_{\mathcal{V}}+\sigma^{2}_{\epsilon}}\,\mu_{\mathcal{V}}\,,$
(2.13)

and applying the law of iterated expectations to
obtain555$\mathcal{E}_{(r)}\equiv\mathcal{E}_{\mathcal{I}(r)}$, as we rank the
business propositions by their estimated values.

$\displaystyle\mathbb{E}(\mathcal{V}_{\mathcal{I}(r)})=\mathbb{E}\left(\mathbb{E}(\mathcal{V}_{\mathcal{I}(r)}\,|\,\mathcal{E}_{(r)})\right)\approx\mu_{\mathcal{V}}+\frac{\sigma^{2}_{\mathcal{V}}}{\sqrt{\sigma^{2}_{\mathcal{V}}+\sigma^{2}_{\epsilon}}}\;\Phi^{-1}\left(\frac{r-c}{N-2c+1}\right)\,.$
(2.14)

Here, decreasing the estimation noise $\sigma^{2}_{\epsilon}$ will lead to an
increase in $\mathbb{E}(\mathcal{V}_{\mathcal{I}(r)})$ for any
${r>\frac{N+1}{2}}$.666While $\mathcal{V}_{\mathcal{I}(r)}$ are no longer
normally distributed with the ranking information, it remains valid to use the
law of iterated expectations to obtain the expected value for all $r$.

The value of business propositions chosen ($\mathcal{W}$) under normal
assumptions then evaluates to

$\displaystyle\mathbb{E}(\mathcal{W})\approx\mu_{\mathcal{V}}+\frac{\sigma^{2}_{\mathcal{V}}}{\sqrt{\sigma^{2}_{\mathcal{V}}+\sigma^{2}_{\epsilon}}}\frac{1}{M}\sum\limits_{r=N-M+1}^{N}\Phi^{-1}\left(\frac{r-c}{N-2c+1}\right)\,.$
(2.15)

This is done by substituting Equation (2.14) into Equation (2.10). Note the
absence of $\mu_{\epsilon}$ in Equation (2.15), which suggests that systematic
bias in estimation will not affect the true value of the chosen business
propositions in the normal case.

Finally, the expression for the expected value of $\mathcal{D}$ when we reduce
the estimation noise from $\sigma^{2}_{\epsilon}=\sigma^{2}_{1}$ to
$\sigma^{2}_{2}$ is much neater under normal assumptions, as many terms cancel
out in Equation (2.11), leading to

$\displaystyle\mathbb{E}(\mathcal{D})\approx\left(\frac{\sigma^{2}_{\mathcal{V}}}{\sqrt{\sigma^{2}_{\mathcal{V}}+\sigma^{2}_{2}}}-\frac{\sigma^{2}_{\mathcal{V}}}{\sqrt{\sigma^{2}_{\mathcal{V}}+\sigma^{2}_{1}}}\right)\frac{1}{M}\sum\limits_{r=N-M+1}^{N}\Phi^{-1}\left(\frac{r-c}{N-2c+1}\right).$
(2.16)

If we further assume that $\mu_{\mathcal{V}}=0$ (i.e., the true value of the
business propositions centres around zero), then the relative gain is entirely
dependent on $\sigma^{2}_{\mathcal{V}}$, $\sigma^{2}_{1}$, and
$\sigma^{2}_{2}$:

$\displaystyle\frac{\mathbb{E}(\mathcal{D}\,|\,\mu_{\mathcal{V}}=0)}{\mathbb{E}(\mathcal{W}\,|\,\sigma^{2}_{\epsilon}=\sigma^{2}_{1},\,\mu_{\mathcal{V}}=0)}=\,\frac{\sqrt{\sigma^{2}_{\mathcal{V}}+\sigma^{2}_{1}}}{\sqrt{\sigma^{2}_{\mathcal{V}}+\sigma^{2}_{2}}}-1\,.$
(2.17)

To calculate the relative improvement in prioritisation delivered by DEM under
these assumptions, we plug the following into Equation (2.17) to obtain an
estimate of how much one will gain from acquiring such capabilities:

  1. 1.

The estimated spread of the values ($\sigma^{2}_{\mathcal{V}}$),

  2. 2.

The estimated deviation of the current estimation process ($\sigma^{2}_{1}$),
and

  3. 3.

The estimated deviation to the actual value upon acquisition of DEM
capabilities ($\sigma^{2}_{2}$).

For example, if Example Company Ltd’s project values are spread with a
standard deviation of 1 unit and their current estimation has a standard error
of 0.5 units, then by acquiring an A/B test framework that is capable of
measuring with an error of 0.4 units, the company gains 3.8% of extra value
simply due to the ability to prioritise with more accurate measurements under
normal assumptions.

We conclude this section by suggesting how one may estimate
$\sigma^{2}_{\mathcal{V}}$, $\sigma^{2}_{1}$, and $\sigma^{2}_{2}$, especially
when they have yet to build any DEM capabilities. Clearly, it is impossible to
recommend specific values for the three parameters as organisations seeking to
build DEM capabilities come in all shapes and sizes. That said, one may
consider using industry averages published in meta-analyses [30, 136] when
estimating $\sigma^{2}_{\mathcal{V}}$. For $\sigma^{2}_{1}$, they may solicit
value estimates for several business propositions from current decision makers
and take the variance over such estimates.777It does not matter if the
decision makers provide value estimates far from the actual value of the
business propositions. In fact, one need not know the business propositions’
actual value. What matters here is the spread of the estimates. Finally, they
may leverage the user numbers and the variance of user responses that form the
business metric to estimate $\sigma^{2}_{2}$ under an A/B test.888For example,
in an A/B test measuring conversion rate (CVR) uplift using a practical
$t$-test, the estimation noise ($\sigma^{2}_{\epsilon}$ or $\sigma^{2}_{2}$)
is the variance of the difference in CVR between two variants under a nil null
hypothesis. This equals $2\cdot\frac{p(1-p)}{n/2}=\frac{4p(1-p)}{n}$, where
$p$ is the base CVR and $n$ is the total number of users across both groups.
See Chapter 3 for further discussions on statistical tests, particularly
Sections 3.5.2 and 3.5.3 on practical $t$-tests and how they are applied to
business metrics based on binary responses such as CVR.  We also encourage one
to explore parameter values around their estimates for robustness, as we will
show in Section 2.7 when we provide two case studies.

###  2.5 Calculating the Variance

It is crucial to understand both the expected gain and the risk or uncertainty
of ranking under lower uncertainty to make effective investment decisions.
Therefore, having derived the expected value in Equation (2.11), we address
the investment risk given by the variance of $\mathcal{D}$ in this section.

The variance calculation features new challenges in addition to that
identified in the section above, the most prominent of which concerns the
interactions between quantities generated under different estimation noise
levels. While these interactions do not affect the expected value, they
influence the variance via the covariance terms. Failure to account for the
covariance terms may lead to a large error in the variance estimate.

Figure 2.3: The generative model in the ranking under lower uncertainty
problem when two distinct noise levels are involved in plate notation. When we
change the noise level of our ranking under uncertainty setup from
$\sigma^{2}_{1}$ to $\sigma^{2}_{2}$ (see Figure 2.2), we obtain two sets of
observed values, $\mathcal{H}_{i}$ and $\mathcal{L}_{i}$, for each noise
level.

To address the challenges, we first extend the notation to clarify the
interactions. Instead of a single set of noise shown in Section 2.3, we define
two sets of random noise with different noise levels:

$\displaystyle\epsilon_{1i}\overset{\textrm{i.i.d.}}{\sim}F_{\epsilon_{1}}(\cdot)\,,$
$\displaystyle\textrm{ where
}\mathbb{E}(\epsilon_{1i})=\mu_{\epsilon}\,,\textrm{Var}(\epsilon_{1i})=\sigma^{2}_{1}\,,\textrm{
and}$
$\displaystyle\epsilon_{2i}\overset{\textrm{i.i.d.}}{\sim}F_{\epsilon_{2}}(\cdot)\,,$
$\displaystyle\textrm{ where
}\mathbb{E}(\epsilon_{2i})=\mu_{\epsilon}\,,\textrm{Var}(\epsilon_{2i})=\sigma^{2}_{2}\,.$
(2.18)

Similar to how the problem is modelled in Section 2.3, we assume all true item
values ($\mathcal{V}$) and noises ($\epsilon_{1}$ and $\epsilon_{2}$) are
independent, i.e., $\mathcal{V}_{i}\perp\epsilon_{1j}$,
$\mathcal{V}_{i}\perp\epsilon_{2j}$, and $\epsilon_{1i}\perp\epsilon_{2j}$
$\forall\,i,j$. The estimated value of each item is then given by

$\displaystyle\mathcal{H}_{i}=\mathcal{V}_{i}+\epsilon_{1i}$
$\displaystyle\quad\textrm{ under {H}igher noise level }\sigma^{2}_{1}\textrm{
and }$ $\displaystyle\mathcal{L}_{i}=\mathcal{V}_{i}+\epsilon_{2i}$
$\displaystyle\quad\textrm{ under {L}ower noise level }\sigma^{2}_{2}.$ (2.19)

The setup is illustrated in Figure 2.3.

Having obtained two sets of estimated values, we rank and trace the
corresponding indices for each set separately. For $\mathcal{H}$, we denote
$\mathcal{H}_{(r)}$ as the $r^{\textrm{th}}$ order statistic of
$\mathcal{H}_{i}$, the estimated value of the $r^{\textrm{th}}$ ranked item
under noise level $\sigma^{2}_{1}$, followed by $\mathcal{V}_{\mathcal{I}(r)}$
as the concomitant [56] of $\mathcal{H}_{(r)}$, i.e., the true value of the
$r^{\textrm{th}}$ item ranked by its estimated value. We repeat the process
for $\mathcal{L}$: we denote $\mathcal{L}_{(s)}$ as the $s^{\textrm{th}}$
order statistic of $\mathcal{L}_{i}$ and $\mathcal{V}_{\mathcal{J}(s)}$ as the
concomitant of $\mathcal{L}_{(s)}$.999$\mathcal{I}(\cdot)$ and
$\mathcal{J}(\cdot)$ are the index functions that map the ranking to the index
for $\mathcal{H}$ and $\mathcal{L}$, respectively.

We also define the mean true value of the top $M$ items, ranked by their
estimated value, under both noise levels as

$\displaystyle\mathcal{W}_{1}=\frac{1}{M}\sum_{r=N-M+1}^{N}\mathcal{V}_{\mathcal{I}(r)}\,,\quad\mathcal{W}_{2}=\frac{1}{M}\sum_{s=N-M+1}^{N}\mathcal{V}_{\mathcal{J}(s)}\,,$
(2.20)

where $\mathcal{W}_{1}$ is the mean true value under $\sigma^{2}_{1}$, and
$\mathcal{W}_{2}$ is the mean true value under $\sigma^{2}_{2}$. Finally, we
denote the difference between the mean true values as
$\mathcal{D}\triangleq\mathcal{W}_{2}-\mathcal{W}_{1}$.

Figure 2.4: Relationship between different variances/covariances used to
calculate the variance of $\mathcal{D}$, the value gained when the estimation
noise is reduced. An arrow from quantity A to B means the value of B depends
on the value of A.

Deriving the variance is similar to deriving the expectation – one has to
obtain the variances for (in order) $\mathcal{H}_{(r)}$/$\mathcal{L}_{(s)}$,
$\mathcal{V}_{\mathcal{I}(r)}$/$\mathcal{V}_{\mathcal{J}(s)}$,
$\mathcal{W}_{1}$/$\mathcal{W}_{2}$, and $\mathcal{D}$. The relationship
between these quantities is shown in Figure 2.4.

####  2.5.1
$\textrm{Var}(\mathcal{H}_{(r)})$/$\textrm{Var}(\mathcal{L}_{(s)})$

We apply a result from [54], which states that the variance of
$\mathcal{H}_{(r)}$ and $\mathcal{L}_{(s)}$ can be approximated as

$\displaystyle\textrm{Var}(\mathcal{H}_{(r)})$
$\displaystyle\approx\frac{r(N-r+1)}{(N+1)^{2}(N+2)}\frac{1}{\left(f_{\mathcal{H}}\big{(}F_{\mathcal{H}}^{-1}\big{(}\frac{r}{N+1}\big{)}\big{)}\right)^{2}}\,,$
(2.21) $\displaystyle\textrm{Var}(\mathcal{L}_{(s)})$
$\displaystyle\approx\frac{s(N-s+1)}{(N+1)^{2}(N+2)}\frac{1}{\left(f_{\mathcal{L}}\big{(}F_{\mathcal{L}}^{-1}\big{(}\frac{s}{N+1}\big{)}\big{)}\right)^{2}}\,,$
(2.22)

where $f(\cdot)$ and $F^{-1}(\cdot)$ denote the probability density function
and quantile function of the corresponding r.v., respectively. As is the case
with calculating the expected value (see Section 2.4), the formula results
from applying the first-order delta method [171], with the same strength and
weakness considerations. In the special case where $\mathcal{V}_{i}$,
$\epsilon_{1i}$, and $\epsilon_{2i}$ are all normally distributed, the
variances are

$\displaystyle\textrm{Var}(\mathcal{H}_{(r)})$
$\displaystyle\approx\frac{r(N-r+1)}{(N+1)^{2}(N+2)}\frac{\sigma^{2}_{X}+\sigma^{2}_{1}}{\big{(}\phi\big{(}\Phi^{-1}\big{(}\frac{r}{N+1}\big{)}\big{)}\big{)}^{2}}\,,$
(2.23) $\displaystyle\textrm{Var}(\mathcal{L}_{(s)})$
$\displaystyle\approx\frac{s(N-s+1)}{(N+1)^{2}(N+2)}\frac{\sigma^{2}_{X}+\sigma^{2}_{2}}{\big{(}\phi\big{(}\Phi^{-1}\big{(}\frac{s}{N+1}\big{)}\big{)}\big{)}^{2}}\,,$
(2.24)

where $\phi(\cdot)$ is the probability density function, and
$\Phi^{-1}(\cdot)$ is the quantile function of a standard normal distribution.

####  2.5.2
$\textrm{Var}(\mathcal{V}_{\mathcal{I}(r)})$/$\textrm{Var}(\mathcal{V}_{\mathcal{J}(s)})$

The variance for $\mathcal{V}_{\mathcal{I}(r)}$ is obtained using properties
of the concomitants of order statistics [55]:101010$\mathcal{H}/\mathcal{L}$
and $\mathcal{V}$ in this thesis correspond to $X$ and $Y$ in [55].

$\displaystyle\textrm{Var}(\mathcal{V}_{\mathcal{I}(r)})=\;$
$\displaystyle\sigma^{2}_{\mathcal{V}}\left(\rho_{\mathcal{V}\mathcal{H}}^{2}\frac{\textrm{Var}(\mathcal{H}_{(r)})}{\sigma^{2}_{\mathcal{V}}+\sigma^{2}_{1}}+1-\rho_{\mathcal{V}\mathcal{H}}^{2}\right)=\frac{\sigma^{2}_{1}\sigma^{2}_{\mathcal{V}}}{\sigma^{2}_{\mathcal{V}}+\sigma^{2}_{1}}+\frac{\sigma^{4}_{\mathcal{V}}}{\left(\sigma^{2}_{\mathcal{V}}+\sigma^{2}_{1}\right)^{2}}\,\textrm{Var}(\mathcal{H}_{(r)})\,,$
(2.25) $\displaystyle\textrm{Var}(\mathcal{V}_{\mathcal{J}(s)})=\;$
$\displaystyle\sigma^{2}_{\mathcal{V}}\left(\rho_{\mathcal{V}\mathcal{L}}^{2}\frac{\textrm{Var}(\mathcal{L}_{(s)})}{\sigma^{2}_{\mathcal{V}}+\sigma^{2}_{2}}+1-\rho_{\mathcal{V}\mathcal{L}}^{2}\right)=\frac{\sigma^{2}_{2}\sigma^{2}_{\mathcal{V}}}{\sigma^{2}_{\mathcal{V}}+\sigma^{2}_{2}}+\frac{\sigma^{4}_{\mathcal{V}}}{\left(\sigma^{2}_{\mathcal{V}}+\sigma^{2}_{2}\right)^{2}}\,\textrm{Var}(\mathcal{L}_{(s)})\,,$
(2.26)

where
$\rho_{\mathcal{V}\mathcal{H}}={\sigma_{\mathcal{V}}}/{\sqrt{\sigma^{2}_{\mathcal{V}}+\sigma^{2}_{1}}}$
denotes the correlation between $\mathcal{V}_{i}$ and $\mathcal{H}_{i}$, and
$\rho_{\mathcal{V}\mathcal{L}}={\sigma_{\mathcal{V}}}/{\sqrt{\sigma^{2}_{\mathcal{V}}+\sigma^{2}_{2}}}$
denotes the correlation between $\mathcal{V}_{i}$ and $\mathcal{L}_{i}$.

In the multivariate normal case, the same result can also be obtained from
first principles. We first recall Equation (2.13), which describes the mean of
$\mathcal{V}_{i}$ once we observe $\mathcal{E}_{i}$ (now $\mathcal{H}_{i}$ or
$\mathcal{L}_{i}$), and note its variance counterpart is

$\displaystyle\sigma^{2}_{\mathcal{V}_{i}\,|\,\mathcal{H}_{i}}=\frac{\sigma^{2}_{1}\sigma^{2}_{\mathcal{V}}}{\sigma^{2}_{\mathcal{V}}+\sigma^{2}_{1}}\,,\quad\sigma^{2}_{\mathcal{V}_{i}\,|\,\mathcal{L}_{i}}=\frac{\sigma^{2}_{2}\sigma^{2}_{\mathcal{V}}}{\sigma^{2}_{\mathcal{V}}+\sigma^{2}_{2}}\,.$
(2.27)

Using the results in Equations (2.13) and (2.27), we apply the law of total
variance to obtain111111Similar to what has been mentioned in Footnotes 5 and
6, $\mathcal{H}_{(r)}\equiv\mathcal{H}_{\mathcal{I}(r)}$ and
$\mathcal{L}_{(s)}\equiv\mathcal{L}_{\mathcal{J}(s)}$, since we rank the
business propositions by their estimated values. Moreover, while
$\mathcal{V}_{\mathcal{I}(r)}$ and $\mathcal{V}_{\mathcal{J}(s)}$ are no
longer normally distributed with the ranking information, using the law of
total variance to obtain the variance for all $r$ and $s$ remains valid.

$\displaystyle\textrm{Var}(\mathcal{V}_{\mathcal{I}(r)})=\,$
$\displaystyle\mathbb{E}\big{(}\textrm{Var}(\mathcal{V}_{\mathcal{I}(r)}\,|\,\mathcal{H}_{(r)})\big{)}+\textrm{Var}\big{(}\mathbb{E}(\mathcal{V}_{\mathcal{I}(r)}\,|\,\mathcal{H}_{(r)})\big{)}$
$\displaystyle\approx\,$
$\displaystyle\frac{\sigma^{2}_{1}\sigma^{2}_{\mathcal{V}}}{\sigma^{2}_{\mathcal{V}}+\sigma^{2}_{1}}+\frac{\sigma^{4}_{\mathcal{V}}}{\sigma^{2}_{\mathcal{V}}+\sigma^{2}_{1}}\frac{r(N-r+1)}{(N+1)^{2}(N+2)}\frac{1}{\big{(}\phi\big{(}\Phi^{-1}(\frac{r}{N+1})\big{)}\big{)}^{2}}\,,$
(2.28) $\displaystyle\textrm{Var}(\mathcal{V}_{\mathcal{J}(s)})=\,$
$\displaystyle\mathbb{E}\big{(}\textrm{Var}(\mathcal{V}_{\mathcal{J}(s)}\,|\,\mathcal{L}_{(s)})\big{)}+\textrm{Var}\big{(}\mathbb{E}(\mathcal{V}_{\mathcal{J}(s)}\,|\,\mathcal{L}_{(s)})\big{)}$
$\displaystyle\approx\,$
$\displaystyle\frac{\sigma^{2}_{2}\sigma^{2}_{\mathcal{V}}}{\sigma^{2}_{\mathcal{V}}+\sigma^{2}_{2}}+\frac{\sigma^{4}_{\mathcal{V}}}{\sigma^{2}_{\mathcal{V}}+\sigma^{2}_{2}}\frac{s(N-s+1)}{(N+1)^{2}(N+2)}\frac{1}{\big{(}\phi\big{(}\Phi^{-1}(\frac{s}{N+1})\big{)}\big{)}^{2}}\,.$
(2.29)

####  2.5.3 $\textrm{Var}(\mathcal{W}_{1})$/$\textrm{Var}(\mathcal{W}_{2})$

To derive the variance of $\mathcal{W}_{1}$, we require the covariance between
$\mathcal{H}_{(r)}$ and $\mathcal{H}_{(s)}$, as well as that between
$\mathcal{V}_{\mathcal{I}(r)}$ and $\mathcal{V}_{\mathcal{I}(s)}$ $\forall
r,s$. The same goes for $\mathcal{W}_{2}$, where we require the covariance
between $\mathcal{L}_{(r)}$ and $\mathcal{L}_{(s)}$, as well as that between
$\mathcal{V}_{\mathcal{J}(r)}$ and $\mathcal{V}_{\mathcal{J}(s)}$ $\forall
r,s$. This is necessary as the terms of $\mathcal{W}_{1}$ (see Equation
(2.2)), which result from removing noise from successive order statistics, are
highly correlated.

Equation 4.6.5 of [56] provided a formula to estimate the covariance between
$\mathcal{H}_{(r)}$ and $\mathcal{H}_{(s)}$ and between $\mathcal{L}_{(r)}$
and $\mathcal{L}_{(s)}$ for any ${r<s\leq N}$:121212For $r>s$, simply swap $r$
and $s$ as covariance functions are symmetrical.

$\displaystyle\textrm{Cov}(\mathcal{H}_{(r)},\mathcal{H}_{(s)})$
$\displaystyle\,\approx\frac{r(N-s+1)}{(N+1)^{2}(N+2)}\frac{1}{f_{\mathcal{H}}\big{(}F_{\mathcal{H}}^{-1}(\frac{r}{N+1})\big{)}\,f_{\mathcal{H}}\big{(}F_{\mathcal{H}}^{-1}(\frac{s}{N+1})\big{)}}\,,$
(2.30) $\displaystyle\textrm{Cov}(\mathcal{L}_{(r)},\mathcal{L}_{(s)})$
$\displaystyle\,\approx\frac{r(N-s+1)}{(N+1)^{2}(N+2)}\frac{1}{f_{\mathcal{L}}\big{(}F_{\mathcal{L}}^{-1}(\frac{r}{N+1})\big{)}\,f_{\mathcal{L}}\big{(}F_{\mathcal{L}}^{-1}(\frac{s}{N+1})\big{)}}\,.$
(2.31)

To obtain the covariance between $\mathcal{V}_{\mathcal{I}(r)}$ and
$\mathcal{V}_{\mathcal{I}(s)}$ for any $r,s\leq N$, we again refer to [55]
(Equation 2.3d):

$\displaystyle\textrm{Cov}(\mathcal{V}_{\mathcal{I}(r)},\mathcal{V}_{\mathcal{I}(s)})$
$\displaystyle=\rho^{2}_{\mathcal{V}\mathcal{H}}\sigma^{2}_{\mathcal{V}}\,\frac{\textrm{Cov}(\mathcal{H}_{(r)},\mathcal{H}_{(s)})}{\sigma^{2}_{\mathcal{V}}+\sigma^{2}_{1}}\,=\frac{\sigma^{4}_{\mathcal{V}}}{\left(\sigma^{2}_{\mathcal{V}}+\sigma^{2}_{1}\right)^{2}}\,\textrm{Cov}(\mathcal{H}_{(r)},\mathcal{H}_{(s)})\,,$
(2.32)
$\displaystyle\textrm{Cov}(\mathcal{V}_{\mathcal{J}(r)},\mathcal{V}_{\mathcal{J}(s)})$
$\displaystyle=\rho^{2}_{\mathcal{V}\mathcal{L}}\sigma^{2}_{\mathcal{V}}\,\frac{\textrm{Cov}(\mathcal{L}_{(r)},\mathcal{L}_{(s)})}{\sigma^{2}_{\mathcal{V}}+\sigma^{2}_{2}}\,=\frac{\sigma^{4}_{\mathcal{V}}}{\left(\sigma^{2}_{\mathcal{V}}+\sigma^{2}_{2}\right)^{2}}\,\textrm{Cov}(\mathcal{L}_{(r)},\mathcal{L}_{(s)})\,.$
(2.33)

Equation (2.32) and (2.33) affirms the claim that
$\mathcal{V}_{\mathcal{I}(r)}$ are positively correlated $\forall r$. Unlike
$\mathcal{V}_{i}$, which are independent by definition, they become correlated
under the presence of ranking information.

We can now state the variance of $\mathcal{W}_{1}$ and $\mathcal{W}_{2}$.
Applying the variance function to Equation (2.20), we get

$\displaystyle\textrm{Var}(\mathcal{W}_{1})=\,$
$\displaystyle\frac{1}{M^{2}}\left(\sum_{r=N-M+1}^{N}\textrm{Var}\left(\mathcal{V}_{\mathcal{I}(r)}\right)\,+\sum_{r=N-M+1}^{N}\,\sum_{s=r+1}^{N}2\cdot\textrm{Cov}\left(\mathcal{V}_{\mathcal{I}(r)},\mathcal{V}_{\mathcal{I}(s)}\right)\right),$
(2.34) $\displaystyle\textrm{Var}(\mathcal{W}_{2})=\,$
$\displaystyle\frac{1}{M^{2}}\left(\sum_{s=N-M+1}^{N}\textrm{Var}\left(\mathcal{V}_{\mathcal{J}(s)}\right)\,+\sum_{r=N-M+1}^{N}\,\sum_{s=r+1}^{N}2\cdot\textrm{Cov}\left(\mathcal{V}_{\mathcal{J}(r)},\mathcal{V}_{\mathcal{J}(s)}\right)\right),$
(2.35)

where the constituent variances and covariances are derived in Equations
(2.25)/(2.26) and (2.32)/(2.33), respectively.

####  2.5.4 $\textrm{Var}(\mathcal{D})$

Finally, we derive the variance of $D$. In addition to the variance of
$\mathcal{W}_{1}$ and $\mathcal{W}_{2}$ derived, respectively, in Equations
(2.34) and (2.35), we require the covariance between these two terms. This, in
turn, requires the covariance between $\mathcal{H}_{(r)}$ and
$\mathcal{L}_{(s)}$ and that between $X_{\mathcal{I}(r)}$ and
$X_{\mathcal{J}(s)}$.

The covariance between $\mathcal{H}_{(r)}$ and $\mathcal{L}_{(s)}$ can be
derived using results in [56]:

$\displaystyle\textrm{Cov}(\mathcal{H}_{(r)},\mathcal{L}_{(s)})=\,$
$\displaystyle\rho_{\mathcal{V}\mathcal{H}}\rho_{\mathcal{V}\mathcal{L}}\,\textrm{Cov}(\mathcal{V}_{(r)},\mathcal{V}_{(s)})$
$\displaystyle=\,$
$\displaystyle\frac{\sigma^{2}_{\mathcal{V}}}{\sqrt{\sigma^{2}_{\mathcal{V}}+\sigma^{2}_{1}}\sqrt{\sigma^{2}_{\mathcal{V}}+\sigma^{2}_{2}}}\frac{r(N-s+1)}{(N+1)^{2}(N+2)}\frac{1}{f_{\mathcal{V}}(F^{-1}_{\mathcal{V}}(\frac{r}{N+1}))\,f_{\mathcal{V}}(F^{-1}_{\mathcal{V}}(\frac{s}{N+1}))}\,,$
(2.36)

where $f_{\mathcal{V}}(\cdot)$ and $F^{-1}_{\mathcal{V}}(\cdot)$ are the
probability density function and quantile function for $\mathcal{V}_{i}$,
respectively.

Deriving the covariance between $\mathcal{V}_{\mathcal{I}(r)}$ and
$\mathcal{V}_{\mathcal{J}(s)}$ is perhaps the most challenging problem within
the work, as they take two forms depending on the indices:

$\displaystyle\textrm{Cov}(\mathcal{V}_{\mathcal{I}(r)},\mathcal{V}_{\mathcal{J}(s)})=\begin{cases}\begin{array}[]{ll}\textrm{Var}(\mathcal{V}_{\mathcal{I}(r)})=\textrm{Var}(\mathcal{V}_{\mathcal{J}(s)})&\textrm{if
}\mathcal{I}(r)=\mathcal{J}(s)\\\
\frac{\sigma^{2}_{\mathcal{V}}}{\sigma^{2}_{\mathcal{V}}+\sigma^{2}_{1}}\frac{\sigma^{2}_{\mathcal{V}}}{\sigma^{2}_{\mathcal{V}}+\sigma^{2}_{2}}\,\textrm{Cov}(\mathcal{H}_{(r)},\mathcal{L}_{(s)})&\textrm{if
}\mathcal{I}(r)\neq\mathcal{J}(s)\end{array},\end{cases}$ (2.37)

where the second case is a standard Bayesian inference result.

The problem arises as the $r^{\textrm{th}}$ ranked $\mathcal{H}$ and the
$s^{\textrm{th}}$ ranked $\mathcal{L}$ can be generated by the same
$\mathcal{V}_{i}$ for some $i$. It will not arise if we have only
$\mathcal{H}$ or $\mathcal{L}$ (see Figure 2.5 for an example). In this case,
when we consider the covariance of the concomitants
$\mathcal{V}_{\mathcal{I}(r)}$/$\mathcal{V}_{\mathcal{J}(s)}$, we have to take
into account both the existing variance of $\mathcal{V}_{i}$ and the ranking
information provided by $\mathcal{H}_{(r)}$ and $\mathcal{L}_{(s)}$. If the
order statistics are generated by different $\mathcal{V}$, we only need to
consider the latter as we assumed $\mathcal{V}_{i}$ to be independent and thus
uncorrelated.

Figure 2.5: Relationship between different quantities in a three-item
generative model under the ranking under lower uncertainty problem.
$\mathcal{V}_{i}$, $\mathcal{H}_{i}$/$\mathcal{L}_{i}$, and
$\mathcal{H}_{(r)}$/$\mathcal{L}_{(s)}$ represent the true value, the unranked
noisy estimates, and the ranked noisy estimates of the items, respectively,
for ${i,r,s\in\\{1,2,3\\}}$. (L) Under one estimation noise level, $\exists$ a
bijection between $\mathcal{V}_{i}$ and $\mathcal{H}_{(r)}$ within a set of
generated samples. (R) With two noise levels, $\mathcal{H}_{(r)}$ and
$\mathcal{L}_{(s)}$ may be generated by the same $\mathcal{V}_{i}$ for some
$r$ and $s$.

As we are interested in the overall behaviour, we only need to derive the two
cases on the RHS of Equation (2.37) and weigh them using the probability that
$\mathcal{I}(r)=\mathcal{J}(s)$ without worrying about which case applies to
each $(r,s)$ pair. The first case (when $\mathcal{I}(r)=\mathcal{J}(s)$) can
be evaluated using the law of total variance with multiple conditioning random
variables:

$\displaystyle\textrm{Var}(\mathcal{V}_{\mathcal{I}(r)})=\textrm{Var}(\mathcal{V}_{\mathcal{J}(s)})$
$\displaystyle=\,$
$\displaystyle\mathbb{E}(\textrm{Var}(\mathcal{V}_{\mathcal{I}(r)}|\mathcal{H}_{(r)},\mathcal{L}_{(s)}))\,+\mathbb{E}(\textrm{Var}(\mathbb{E}(\mathcal{V}_{\mathcal{I}(r)}|\mathcal{H}_{(r)},\mathcal{L}_{(s)})|\mathcal{H}_{(r)}))+\textrm{Var}(\mathbb{E}(\mathcal{V}_{\mathcal{I}(r)}|\mathcal{H}_{(r)}))$
$\displaystyle=\,$
$\displaystyle\frac{\sigma^{2}_{\mathcal{V}}\sigma^{2}_{1}\sigma^{2}_{2}}{\sigma^{2}_{\mathcal{V}}\sigma^{2}_{1}+\sigma^{2}_{\mathcal{V}}\sigma^{2}_{2}+\sigma^{2}_{1}\sigma^{2}_{2}}\,+\left(\frac{\sigma^{2}_{\mathcal{V}}\sigma^{2}_{1}}{\sigma^{2}_{\mathcal{V}}\sigma^{2}_{1}+\sigma^{2}_{\mathcal{V}}\sigma^{2}_{2}+\sigma^{2}_{1}\sigma^{2}_{2}}\right)^{2}\textrm{Var}(\mathcal{L}_{(s)})\,+\left(\frac{\sigma^{2}_{\mathcal{V}}}{\sigma^{2}_{\mathcal{V}}+\sigma^{2}_{1}}\right)^{2}\textrm{Var}(\mathcal{H}_{(r)}).$
(2.38)

The second case can be derived by substituting Equation (2.36) into Equation
(2.37).

For the weighting probability $\mathbb{P}(\mathcal{I}(r)=\mathcal{J}(s))$, we
see its derivation as an interesting and potentially important problem in its
own right, yet to the best of our knowledge, no proper treatment was given to
the problem. In this work, we approximate the probability using beta-binomial
distributions, with parameters derived from quantities calculated above.
Without distracting readers from the main question of quantifying the value
and risk of DEM capabilities, we relegate the detailed discussion on
approximating the quantity to Appendix A.1.

With the three components for the covariance between
$\mathcal{V}_{\mathcal{I}(r)}$ and $\mathcal{V}_{\mathcal{J}(s)}$ in place, we
can finally derive $\textrm{Cov}(\mathcal{W}_{1},\mathcal{W}_{2})$ and
$\textrm{Var}(\mathcal{D})$ by applying the covariance and variance functions
to the definitions of, respectively, $\mathcal{W}_{1}$/$\mathcal{W}_{2}$ and
$\mathcal{D}$ (see (2.20)) to obtain

$\displaystyle\textrm{Cov}(\mathcal{W}_{1},\mathcal{W}_{2})=\,$
$\displaystyle\frac{1}{M^{2}}\sum_{r=N-M+1}^{N}\sum_{s=N-M+1}^{N}\textrm{Cov}(\mathcal{V}_{\mathcal{I}(r)},\mathcal{V}_{\mathcal{J}(s)}),$
(2.39) $\displaystyle\textrm{Var}(\mathcal{W})=\,$
$\displaystyle\textrm{Var}(\mathcal{W}_{2}-\mathcal{W}_{1})=\textrm{Var}(\mathcal{W}_{1})+\textrm{Var}(\mathcal{W}_{2})-2\,\textrm{Cov}(\mathcal{W}_{1},\mathcal{W}_{2}),$
(2.40)

where the first two terms on the RHS of Equation (2.40) are that derived in
Equations (2.34) and (2.35).

We conclude this section by observing that $M$ and $N$ influence
$\textrm{Var}(\mathcal{D})$ considerably. In particular,
$\textrm{Var}(\mathcal{D})$ is generally large when $M$ and $N$ are small with
other parameters fixed. This is crucial as even in cases where
$\mathbb{E}(\mathcal{D})$ is positive, the limited capacity of an organisation
to introduce new business propositions may mean that the Sharpe ratio defined
in Section 2.3 (see Expression (2.4)) may not be high enough to justify
investment in a DEM capability.

The exact threshold where an organisation should consider acquiring such
capabilities depends on multiple factors, including their size (which affects
$M$), the size of their backlog ($N$), the nature of their work
($\mu_{\mathcal{V}}$ and $\sigma^{2}_{\mathcal{V}}$), and how good they were
at estimation ($\sigma^{2}_{1}$). Thus, we refrain from providing a one-size-
fits-all recommendation but give examples in Section 2.7.

###  2.6 Empirical Verification

Having performed theoretical calculations for the expectation and variance of
the value DEM capabilities deliver through enhanced prioritisation, we verify
those calculations using simulation results.131313All code used in the
experiments, case studies and extensions are available on GitHub:
https://github.com/liuchbryan/ranking_under_lower_uncertainty.

We verify the result derived in Sections 2.4 and 2.5 empirically, in
particular under the normal assumptions. We run multiple _statistical tests_
for each quantity of interest – the mean and variance of
$\mathcal{W}_{1}$/$\mathcal{W}_{2}$ and $\mathcal{D}$, as well as the
covariance between different pairs of order statistics and their concomitants.

In each _statistical test_ , we randomly select and fix the value of the
parameters, i.e., $N$, $M$, $\mu_{\mathcal{V}}$, $\mu_{\epsilon}$,
$\sigma^{2}_{\mathcal{V}}$, $\sigma^{2}_{1}$, $\sigma^{2}_{2}$, $r$, and $s$
(the latter two for the covariance of the order statistics only). We select
$N$ uniformly on a log scale, with the log scale favoured due to its better
resemblance to the organisation size distribution in real life and bounds
chosen to represent the usual number of items/propositions an organisation
will consider during prioritisation. We also select $\mu_{\mathcal{V}}$,
$\mu_{\epsilon}$, $\sigma^{2}_{\mathcal{V}}$, and $\sigma^{2}_{1}$ randomly
from a uniform distribution (on a linear scale), with bounds chosen that are
reflective of the usual business metrics an organisation uses to prioritise
their items/propositions and their unit of measurement (e.g., percentage
points change, revenue impact in hundred thousand pounds).141414Denoting
$U(a,b)$ as a uniform distribution with bounds $a$ and $b$, we draw $N\sim
10^{U(1,3.5)}$, $\mu_{\mathcal{V}},\mu_{\epsilon}\sim U(-10,10)$ and
$\sigma^{2}_{\mathcal{V}},\sigma^{2}_{1}\sim U(0.3,10)^{2}$.

Each parameter is drawn independently from other parameters where possible,
with the only exceptions being when the range of a parameter is restricted
(i.e., ${M<N}$, ${\sigma^{2}_{2}<\sigma^{2}_{1}}$, and ${N-M+1}\leq r,s,\leq
N$ for selected items/propositions). To draw $M$, we draw a uniformly random
multiplier between 0.01 and 0.8, then apply the multiplier to $N$.151515In
other words, we draw $M\sim N\times U(0.01,0.8)$, subjected to rounding with
the floor function and a minimum value of one. Multipliers above 0.8 are
ignored, as it is unrealistic for organisations to have such a high relative
capacity. We apply a similar process to draw $\sigma^{2}_{2}$, though we use a
different multiplier range and apply the multiplier on $\sigma_{1}$ (the
standard deviation) before squaring.161616We draw
$\sigma^{2}_{2}\sim(\sigma_{1}\times U(0.1,0.99))^{2}$, subjected to a minimum
value of $0.2^{2}$. Multipliers below 0.1 and above 0.99 are ignored, as we
consider such an estimation noise reduction unrealistic and of negligible
value, respectively. Finally, we draw $r$ and $s$ independently and uniformly
amongst integers between $N-M+1$ and $N$ (both inclusive).

Upon selecting and fixing the value of the parameters, we compare the
theoretical value of the quantity of interest to the centred 95% confidence
interval (CI) generated from multiple _empirical samples_. If the derivations
above are exact, the 95% CI should contain the theoretical value in around 95%
of the statistical tests. The histogram of the percentile rank of the
theoretical quantity among the empirical samples should also follow a uniform
distribution [259].

Table 2.1: The number of statistical tests with the centred 95% confidence interval containing the derived theoretical value for each quantity of interest. If a theoretical value derived in Sections 2.4 or 2.5 is exact, its associated 95% CI should contain the theoretical value in 95% of the statistical tests. Quantity | # within CI | # statistical tests | % within CI
---|---|---|---
$\mathbb{E}(\mathcal{W})$ | 3,991 | 4,428 | 90.13%
$\mathbb{E}(\mathcal{D})$ | 4,162 | 4,428 | 93.99%
$\textrm{Var}(\mathcal{W})$ | 3,390 | 4,555 | 74.42%
$\textrm{Cov}(\mathcal{H}_{(r)},\mathcal{L}_{(s)})$ | 4,663 | 4,940 | 94.39%
$\textrm{Cov}(\mathcal{V}_{\mathcal{I}(r)},\mathcal{V}_{\mathcal{J}(s)})$ | 4,730 | 4,940 | 95.75%

Each _empirical sample_ is generated using one of the following two methods
depending on the quantity we are evaluating:

a) Bootstrap resampling – We use the method to generate a sample for the
mean/variance of $\mathcal{W}_{1}$/$\mathcal{W}_{2}$ and $\mathcal{D}$. We
first generate the initial samples for $\mathcal{W}_{1}$, $\mathcal{W}_{2}$,
and $\mathcal{D}$ by performing 10,000 _simulation runs_ (see below). We then
resample the initial samples and calculate the mean/variance of the resample
to obtain an empirical mean/variance sample. Finally, we repeat the resampling
2,000 times to obtain a representative empirical distribution for the
mean/variance.

b) Sampling for order statistics – The bootstrapping approach is unlikely to
work on the covariance between the order statistics (e.g., $\mathcal{H}_{(r)}$
and $\mathcal{L}_{(s)}$) and their concomitants (e.g.,
$\mathcal{V}_{\mathcal{I}(r)}$ and $\mathcal{V}_{\mathcal{J}(s)}$), as the
ranking information may not be preserved during resampling. Hence, for these
quantities, we opt for a more naïve sampling approach. We generate 200 samples
for $\mathcal{H}_{(r)}$, $\mathcal{L}_{(s)}$, $\mathcal{V}_{\mathcal{I}(r)}$,
and $\mathcal{V}_{\mathcal{J}(s)}$ via the same number of _simulation runs_ ,
and calculate the covariance between these quantities to obtain an empirical
sample. The process is repeated 1,000 times to yield a representative
empirical distribution for the covariance.

(a) $\mathbb{E}(\mathcal{W})$

(b) $\mathbb{E}(\mathcal{D})$

Figure 2.6: Histogram of the theoretical quantity’s percentile rank compared
to the empirical samples across multiple statistical tests. If the theoretical
value derived in Section 2.4 is exact, the histogram should show a uniform
distribution with probability mass around the black line [259].

Finally, in each _simulation run_ , we obtain one sample for each of
$\mathcal{H}_{(r)}$/$\mathcal{L}_{(s)}$,
$\mathcal{V}_{\mathcal{I}(r)}$/$\mathcal{V}_{\mathcal{J}(s)}$,
$\mathcal{W}_{1}/\mathcal{W}_{2}$, and $D$ w.r.t. the parameters via the
following four-step process:171717Identifiers in monospace refer to variables
used in software packages, which correspond to the random variables used in
Sections 2.3–2.5.

  1. 1.

Take $N$ samples from
$\mathcal{N}(\mu_{\mathcal{V}},\sigma^{2}_{\mathcal{V}})$ as Vi;

  2. 2.

Take $N$ samples from $\mathcal{N}(\mu_{\epsilon},\sigma^{2}_{1})$, and sum
the ith-indexed sample with Vi $\forall$i to obtain Hi:

    1. (a)

Rank all Hi, take the value of the rth-ranked Hi as H(r) and its index as
I(r),

    2. (b)

Take the value of the I(r)th-indexed sample of Vi as VI(r),

    3. (c)

Obtain the indices of the $M$ largest samples of the ranked Hi,

    4. (d)

Calculate the mean of Vi, where i is in the set of indices in Step 2c, as W1;

  3. 3.

Take $N$ samples from $\mathcal{N}(\mu_{\epsilon},\sigma^{2}_{2})$, and sum
the ith-indexed sample with Vi $\forall$i to obtain Li:

    1. (a)

Rank all Li, take the value of the sth-ranked Li as L(s) and its index as
J(s),

    2. (b)

Take the value of the J(s)th-indexed sample of Vi as VJ(s),

    3. (c)

Obtain the indices of the $M$ largest samples of the ranked Li,

    4. (d)

Calculate the mean of Vi, where i is in the set of indices in Step 3c, as W2;

  4. 4.

Take the difference between W2 obtained in Step 3d and W1 from Step 2d to get
D.

We show the results in Table 2.1 and Figure 2.6. We observed that the 95% CI
of the quantities $\mathbb{E}(\mathcal{W})$, $\mathbb{E}(\mathcal{D})$,
$\textrm{Var}(\mathcal{W})$,
$\textrm{Cov}(\mathcal{H}_{(r)},\mathcal{L}_{(s)})$, and
$\textrm{Cov}(\mathcal{V}_{\mathcal{I}(r)},\mathcal{V}_{\mathcal{J}(s)})$
contain the derived theoretical value for roughly 90%, 94%, 74%, 94%, and 96%
of the times respectively. While these numbers are expected for the
expectations and covariances, considering they are approximations, they are on
the low side for the variances. Upon further investigation, we realised that
most out-of-CI cases have a theoretical variance below the CI, suggesting a
slight underestimate in our variance derivation. We believe that this is due
to the omission of higher-order terms when using the formulas in [54], leading
to a small (less than one percent relative) bias, which is more apparent for
smaller $N$ and $M$. Otherwise, we are satisfied with the soundness of the
derived quantities.

###  2.7 Case Study

_“What do e-commerce/marketing companies gain by acquiring digital
experimentation and measurement capabilities?”_

It is difficult to verify any model that seeks to ascertain the value of DEM
capabilities with real-life data. This is because of the inability to observe
the true value of a digital product / business proposition / service without
any measurement error and the lack of published measurements from
organisations. The closest proxies are meta-analyses, including those compiled
by Browne and Johnson [30] and Johnson et al. [136], which contain statistics
on the measured uplift (in relative %) over a large number of e-commerce and
marketing experiments for many organisations.

The information presented by the two groups of researchers, which we describe
in more detail below, is sufficient for us to ask the following question: If
the same organisation conducted all the experiments presented by Browne and
Johnson / Johnson et al., how much value did the DEM capabilities add due to
improved prioritisation? We present results under normal assumptions in this
section and will revisit the question when we discuss the model under
$t$-distributed assumptions in Section 2.8.1.

####  2.7.1 e-Commerce companies

In [30], Browne and Johnson reported running 6,700 A/B tests in e-commerce
companies, with an overall effect in relative conversion rate (CVR) uplift
centred at around zero, and the 5% and 95% percentiles at around
$\pm[1.2\%,1.3\%]$. We then divide the range by ${z_{0.95}\approx 1.645}$, the
95th percentile of a standard normal, to estimate that the distribution
reported has a standard deviation of around 0.75%. Based on this information,
we take $\mu_{\mathcal{V}}=0$ and $\sigma^{2}_{\mathcal{V}}=(0.6\%)^{2}$,
considering that the reported distribution incorporated some estimation noise,
and hence, the spread of the true values should be slightly lower.

Given an A/B test on CVR uplift run by the most prolific organisations (e.g.
one with five million visitors and a 5% CVR) carries an estimation noise of
around $(0.28\%)^{2}$,8 we explore the scenarios where we reduce the noise
level from $\sigma^{2}_{1}=\\{(1\%)^{2},(0.8\%)^{2},(0.6\%)^{2}\\}$ to
$\sigma^{2}_{2}=\\{(0.8\%)^{2},(0.6\%)^{2},(0.4\%)^{2}\\}$, representing
different levels of estimation abilities before and after the acquisition of
DEM capabilities for companies of various sizes. We also calculate the value
gained under different $M$ (from 10 to 2,000) to simulate organisations with
different yet realistic capacities while fixing $N=6700$ (# experiments).
Finally, we set $\mu_{\epsilon}=0$ as we do not assume any systematic bias
during estimation in this case.

We show the results in Figure 2.7, which shows the relationship between
different $M$ and the value gained under different magnitudes of estimation
noise reduction. One can observe that the expected gain in value decreases
when $M$ increases. This is expected: as one increases their capacity, they
will run out of the most valuable work and have to settle for less valuable
work that has many acceptable replacements with similar value, limiting the
value DEM capabilities bring.

We can also see an inverse relationship between the size of $M$ and the
uncertainty of the value gained. As a result, while the expected value gain
decreases with increasing $M$, the uncertainty drops quicker, such that at
some $M$, we will see a statistically significant increase in value gained or
an acceptable Sharpe ratio that justifies investment in DEM capabilities or
both. The specific value that tips the balance is heavily dependent on
individual circumstances.

(a) $(1\%)^{2},(0.8\%)^{2}$

(b) $(1\%)^{2},(0.6\%)^{2}$

(c) $(1\%)^{2},(0.4\%)^{2}$

(d) $(0.8\%)^{2},(0.6\%)^{2}$

(e) $(0.8\%)^{2},(0.4\%)^{2}$

(f) $(0.6\%)^{2},(0.4\%)^{2}$

Figure 2.7: The value gained by having some digital experimentation and
measurement (DEM) capabilities $\mathcal{D}$ (x-axes, in per cent) under
different capacity $M$ ($y$-axes, in log scale) in the case study on 6,700
e-commerce experiments reported by Browne and Johnson [30] (see Section
2.7.1). In each plot, the dot represents the mean, and the error bar
represents the 5th–95th percentile of the empirical value distribution. Each
sub-caption denotes the estimation noise before and after the acquisition of
DEM capabilities (i.e., $\sigma^{2}_{1},\sigma^{2}_{2}$). We fix
$\mu_{\mathcal{V}},\mu_{\epsilon}=0$, $\sigma^{2}_{\mathcal{V}}=(0.6\%)^{2}$,
and $N=6700$.

####  2.7.2 Marketing companies

In the second case study, we repeat the process applied to e-commerce in
Section 2.7.1 for the marketing experiments described in [136]. In that work,
Johnson et al. reported running 184 marketing experiments that measured
relative CVR uplift, with a mean relative uplift of 19.9% and standard error
of 10.8%. Thus, we take $\mu_{\mathcal{V}}=19.9\%$ and
$\sigma^{2}_{\mathcal{V}}=(10\%)^{2}$, which is slightly reduced to account
for the estimation noise being included in the reported standard error.

Johnson et al. also noted that the average sample size in these experiments is
over five million, which keeps the estimation noise low. However, the design
of marketing experiments often comes with additional sources of noise compared
to standard A/B tests [105, 172]. Hence, we assume the same estimation noise
as in the e-commerce case study above (i.e.,
$\sigma^{2}_{2}=\\{(0.8\%)^{2},(0.6\%)^{2},(0.4\%)^{2}\\}$). The larger
variance in the uplifts allows us to assume a larger estimation error without
DEM capabilities, and we explore the scenarios where
$\sigma^{2}_{1}=\\{(5\%)^{2},(2\%)^{2},(1\%)^{2},(0.8\%)^{2},(0.6\%)^{2}\\}$.
We set $N=184$ (# experiments) and vary $M$ between 10 and 100 for each
combination of $\sigma^{2}_{1}$ and $\sigma^{2}_{2}$.

Figure 2.8 shows the results. In the presence of a larger variability in the
true uplift of the advertising campaigns ($\sigma^{2}_{\mathcal{V}}$) and
lower capacity ($M$), the level of estimation noise reduction that gave a
statistically significant value gained in the e-commerce example is no longer
sufficient. Therefore, one needs a larger noise reduction or to increase their
capacity to effectively control the risk in investing in DEM capabilities.
They may also be better off increasing their limited number of existing
business propositions.

(a) $(5\%)^{2},(0.8\%)^{2}$

(b) $(5\%)^{2},(0.6\%)^{2}$

(c) $(5\%)^{2},(0.4\%)^{2}$

(d) $(2\%)^{2},(0.8\%)^{2}$

(e) $(2\%)^{2},(0.6\%)^{2}$

(f) $(2\%)^{2},(0.4\%)^{2}$

(g) $(1\%)^{2},(0.8\%)^{2}$

(h) $(1\%)^{2},(0.6\%)^{2}$

(i) $(1\%)^{2},(0.4\%)^{2}$

(j) $(0.8\%)^{2},(0.6\%)^{2}$

(k) $(0.8\%)^{2},(0.4\%)^{2}$

(l) $(0.6\%)^{2},(0.4\%)^{2}$

Figure 2.8: The value gained by having some digital experimentation and
measurement (DEM) capabilities $\mathcal{D}$ (x-axes, in per cent) under
different capacity $M$ ($y$-axes) in the case study on 184 marketing
experiments reported by Johnson et al. [136] (see Section 2.7.2). In each
plot, the dot represents the mean, and the error bar represents the 5th–95th
percentile of the empirical value distribution. Each sub-caption denotes the
estimation noise before and after the acquisition of DEM capabilities (i.e.,
$\sigma^{2}_{1},\sigma^{2}_{2}$). Here, we fix $\mu_{\mathcal{V}}=19.9\%$,
$\mu_{\epsilon}=0$, $\sigma^{2}_{\mathcal{V}}=(10\%)^{2}$, and $N=184$.

###  2.8 Empirical Extensions

We also provide two extensions, evaluated empirically, that open the door for
future work in this area.

####  2.8.1 Valuation under independent $t$-distributed assumptions

So far, we have spent much of the work assuming that the business
propositions’ true value and the estimation noise are normally distributed.
While possessing decent mathematical properties, more is needed to explain the
heavy tail in the distribution of uplifts shown in [30] or [136].

In this section, we model the true value of the business propositions and the
estimation noise as Generalised Student’s $t$-distributions (see Equation
(2.7)). It is difficult to derive the exact theoretical quantities under such
model assumptions because Student’s $t$-distributions do not have conjugate
priors [230]. We instead simulate the empirical distribution of the value
gained under different parameter combinations to understand if this model is a
better alternative to that under normal assumptions. The sampling procedure is
similar to that described in Section 2.6, with steps modified to generate
samples using standard $t$-distributions, which are then scaled and located as
specified by Equation (2.7).

We compare the value gain estimates obtained under t-distributed and normal
assumptions. For each comparison, we randomly sample values for $N$, $M$,
$\mu_{\mathcal{V}}$, $\mu_{\epsilon}$, $\sigma^{2}_{\mathcal{V}}$,
$\sigma^{2}_{1}$, $\sigma^{2}_{2}$, and perform 1,000 simulation runs of the
four-step sampling procedure in Section 2.6 to obtain samples of D using both
the $t_{3}$ and normal distributions.181818$t_{3}$ ($t$-distribution with
three degrees of freedom (d.f.)) is used as it is the distribution with the
longest tail under the $t$ family with a natural number of d.f. while
retaining a finite variance. We then compare the expected values, the 5th and
95th percentile of the value gained under the two distributions.

(a) e-Commmerce - $(1\%)^{2},(0.4\%)^{2}$

(b) e-Commerce - $(0.6\%)^{2},(0.4\%)^{2}$

(c) Marketing - $(2\%)^{2},(0.4\%)^{2}$

(d) Marketing - $(0.8\%)^{2},(0.4\%)^{2}$

Figure 2.9: The value gained by having some digital experimentation and
measurement capabilities (x-axes, in per cent) under normal assumptions (blue
error bars with crosses) and $t$-distributed assumptions (orange error bars
with triangles). Both assumptions are evaluated under different but matching
capacity $M$ ($y$-axes, with the error bars displaced for clarity). See
Section 2.7.1/Figure 2.7 and Section 2.7.2/Figure 2.8 for details of the
parameters for e-commerce and marketing experiments, respectively.

We observed from 840 comparisons that overall, the value gained under the
$t$-distributed assumptions has a higher mean (7% higher mean) and variance
(7% higher in the 95% percentile on average) than that under normal
assumptions. The result arises despite us setting the mean/variance of the
true value and estimation noise under the $t$-distributed assumptions to that
under the normal assumptions. This suggests that the model under
$t$-distributed assumptions can capture the “higher risk, higher reward”
concept.

Individual comparisons paint a more nuanced picture, perhaps best illustrated
by revisiting the case study in Section 2.7 under $t$-distributed assumptions.
We select a few scenarios featured in the previous section and overlay the
value gained by having DEM capabilities under $t$-distributed assumptions over
that under normal assumptions in Figure 2.9. One can see that while
$t$-distributed assumptions generally yield a higher value gained, this is not
always the case – for the e-commerce case, as $M$ increases, the value gained
decreases quicker under $t$-distributed assumptions than under normal
assumptions. This shows that the valuation of DEM capabilities is sensitive to
model assumptions.

####  2.8.2 Partial estimation/measurement noise reduction

There are many situations when not all business propositions are immediately
measurable upon acquiring DEM capabilities. This may be due to the extra work
required to integrate additional capabilities in certain legacy systems or the
limited ability to run experiments on online but not offline activities. In
the case where there is a single backlog, we ask the question, will an
organisation still benefit from a partial noise reduction when some business
propositions’ values are obtained under reduced uncertainty while others are
subject to the original noise level?

We address this by attempting to establish the relationship between the
expected improvement in the mean true value of the selected business
propositions and the proportion of business propositions that benefited from a
reduced estimation noise (denoted ${p\in[0,1]}$).191919We can model the
estimation noise using a two-component mixture distribution parameterised by
$p$. The sampling procedure is similar to that described in Section 2.6, with
Step 3 modified: instead of generating all samples from
$\mathcal{N}(\mu_{\epsilon},\sigma^{2}_{2})$, we generate $p$ of the samples
from $\mathcal{N}(\mu_{\epsilon},\sigma^{2}_{2})$ (the lowered estimation
noise) and $1-p$ of the samples from
$\mathcal{N}(\mu_{\epsilon},\sigma^{2}_{1})$ (the original estimation noise).

(a) $N=50$, $M=5$

(b) $N=50$, $M=20$

(c) $N=2500$,
$M=250$

(d) $N=2500$,
$M=1000$

(e) $N=50$, $M=5$

(f) $N=50$, $M=20$

(g) $N=2500$,
$M=250$

(h) $N=2500$,
$M=1000$

Figure 2.10: The near-linear relationship between $p$ (proportion of business
propositions with their value estimated under a lower estimation noise,
$x$-axes) and the improvement in the mean true value of the selected business
propositions ($y$-axes) under normal assumptions. In each plot, the dot
represents the sample mean, and the error bar represents the 5%–95% percentile
of the sample value gained. All figures assume $\sigma^{2}_{\mathcal{V}}=1$,
while the top four figures assume $\sigma^{2}_{1}=0.5^{2}$ and
$\sigma^{2}_{2}=0.4^{2}$ (corresponding to a small reduction in estimation
noise), and the bottom four figures assume $\sigma^{2}_{1}=0.8^{2}$ and
$\sigma^{2}_{2}=0.2^{2}$ (corresponding to a large reduction in estimation
noise).

We run the procedure above under various scenarios, including a large/small
$N$, a large/small ratio between an organisation’s capacity and backlog
(${M}/{N}$), and a large/small magnitude of noise reduction upon acquisition
of DEM capabilities ($\sigma^{2}_{1}-\sigma^{2}_{2}$). Figure 2.10 shows the
result. We can see that under most scenarios, the expected value gained
increases with $p$ linearly, while there are a few scenarios where the
expected improvement in mean true value of the selected business propositions
curve upward for increasing $p$. This shows that while organisations are
incentivised to acquire DEM capabilities that cover the majority of their
work, in many scenarios, a partial acquisition yields proportional benefits.
Potential experimenters need not consider the acquisition a zero-one decision
or worry about any steep initial investment required to unlock returns.

###  2.9 A Brief Recap

In this chapter, we have addressed the problem of valuing DEM capabilities,
which enables one to justify acquiring the said capabilities. Such
capabilities deliver three forms of value to organisations. These are

  1. 1.

Improved recognition of the value of business propositions,

  2. 2.

Enhanced capability to prioritise, and

  3. 3.

The ability to optimise individual business propositions.

Of these, improving prioritisation is the most challenging to address while
being the most applicable for organisations seeking to build DEM capabilities
from scratch.

We have established a methodology to value better prioritisation through
reduced estimation error using the framework of ranking under uncertainty. The
key insight is that DEM capabilities reduce the estimation error in the value
of individual business propositions, allowing prioritisation to follow the
optimal order of projects more closely had the true values of business
propositions been observable. In addition, we have provided simple formulas
that give the value of prioritising under lower estimation error brought by
DEM capabilities and the Sharpe ratio governing investment decisions. Finally,
alongside two case studies that illustrate how we can apply the methodology in
specific situations, we have provided general guidelines for conditions when
such investments are inappropriate.

##  Chapter 3 Statistical Testing

This chapter contains background material that appeared in multiple
publications:

  * •

“An Evaluation Framework for Personalization Strategy Experiment Designs”,
presented at and awarded Best Student Paper of AdKDD 2020 Workshop (in
conjunction with SIGKDD ’20) [176];

  * •

“Datasets for Online Controlled Experiments”, presented at the 35th Conference
on Neural Information Processing Systems (NeurIPS 2021) [173]; and

  * •

“Measuring e-Commerce Metric Changes in Online Experiments”, presented at ACM
Web Conference 2023 (WWW ’23) [177].

###  3.1 Motivation

After establishing the business case of engaging in digital experimentation
and measurement, we turn our focus to two necessary ingredients for running
digital experiments. They are data, which record what happens during an
experiment, and statistical tests, which indicate whether groups in the
experiment give different responses beyond randomness.

We first discuss statistical testing in this chapter, which will complement
our approach to datasets in Chapter 4. We will also build upon the concepts
introduced in this chapter in Chapter 5, when we formally describe an
experiment and its underlying causal reasoning, and in Chapter 6, when we
compare different experimental designs for digital experiments. Generally (and
loosely) speaking, a statistical test is a statistical inference procedure
that informs us whether the data collected is compatible with our
hypothesis(es). In the context of digital experimentation, the data collected
is often responses from two or more groups exposed to different treatments,
and the hypothesis is usually whether or how much the responses from one group
are different from another group.

Statistical testing may appear to be a simple and widely understood topic – at
the end of the day, it is taught in many upper secondary education curricula