The table shows several gaps in OCE dataset availability, the most obvious
being the lack of datasets that record responses at a sub-randomisation unit
level. In the sections below, we will identify more of these gaps and discuss
their implications for digital experiment analysis.

Dataset Name Ref. Experiment Count \- Variant / Metric Count Response
Granularity Time Granularity Syntheticity Application Domain Target
Demographic Temporal Coverage Documentation Data management / long- term
preservation plan Upworthy Research Archive [185] Multiple (32 487) \- 2–14 /
2 (C) Group Overall result only Timestamp per expt. Real Media & Ads
Copy/Creative Mostly English- speaking users in USA Jan 2013 – Apr 2015 Peer
reviewed data article Host: Open Science Framework DOI: ✓ (see [185]) Licence:
CC BY 4.0 ASSISTments Dataset from Multiple Randomized Controlled Experiments
[244] Multiple (22) \- 2 / 2 (BC) Rand. Unit Overall result only ✗ timestamp
Real Education Teaching model Mostly middle school students (age 11-14)
in/near MA, USA 2013 – 2015 Peer reviewed data article Host: Author website
DOI: Unknown Licence: Unknown A/B Testing Web Analytics Data (From [292])
[293] Single \- 5 / 2 (C) Group Overall result only $\checkmark$ timestamp
Real Education UX/UI change Mostly English- speaking university library users
May 2013 – Jun 2013 Accompanying dataset to peer-reviewed research article
Host: University library DOI: ✓ (see [293]) Licence: CC BY-SA 4.0 Dataset of
two experiments of the application of gamified peer assessment model into
online learning environment MeuTutor (From [261]) [262] Multiple (2) \- 3+2 /
3+3 (R+C) Rand. Unit Overall result only ✗ timestamp Real Education Teaching
model High school students in Brazil taking ENEM (age 17) Jul 2015 – Aug 2015
Peer reviewed data article Host: Journal website DOI: ✓ (see [262]) Licence:
CC BY 4.0 Udacity Free Trial Screener Experiment (From Udacity A/B Testing
Course - Final Project [111]) See e.g. [205, 247, 273] Single \- 2 / 4 (C)
Group Daily checkpoint Snapshot Real Education UX/UI change Mostly English-
speaking users Unknown Blog posts & Kaggle notebooks e.g. [192, 236, 294]
Host: Kaggle / GitHub (multiple) DOI: Unknown Licence: Unknown “Analyse A/B
Test Results” Dataset (From Udacity Online Data Analyst Course - Project 3
[264]) See e.g. [4, 57, 215] Single \- 2 / 1 (B) Rand. Unit Overall result
only Timestamp per RU Unknown E-commerce UX/UI change Unknown Unknown Blog
posts & Kaggle notebooks e.g. [42, 180, 227] Host: Kaggle / GitHub (multiple)
DOI: Unknown Licence: Unknown Mobile Games A/B Testing with Cookie Cats
(DataCamp project [15]) See e.g. [82, 257, 291] Single \- 2 / 3 (BC) Rand.
Unit Overall result only ✗ timestamp Real Gaming Design change Unknown (likely
Facebook users) Unknown Kaggle notebook Host: Kaggle / Course website DOI:
Unknown Licence: Unknown Experiment Dataset (From DataCamp A/B Testing in R
Course [33]) [35] Single \- 2 / 1 (B) Rand. Unit Overall result only Timestamp
per RU Synthetic Tech UX/UI Change N/A N/A Course notes Blog posts Host:
Course website DOI: Unknown Licence: Unknown Data Visualization Website -
April 2018 (From DataCamp A/B Testing in R Course [33]) [34] Single \- 2 / 4
(BR) Rand. Unit Overall result only Timestamp per RU Synthetic Tech UX/UI
Change N/A N/A Course notes Host: Course website DOI: Unknown Licence: Unknown
AB Testing Result (From Customer Analytics and A/B Testing in Python [113])
[112] Single \- 2 / 2 (CR) Rand. Unit Overall result only Timestamp per RU
Unknown Tech UX/UI Change Unknown Jan 2014 – Jan 2018 Course notes Host:
Course website DOI: Unknown Licence: Unknown Grocery Website Data for AB Test
[147] Single \- 2 / 1 (B) Rand. Unit Overall result only ✗ timestamp Unknown
E-commerce UX/UI change Unknown Unknown Kaggle notebook Host: Kaggle DOI:
Unknown Licence: Unknown Ad A/B Testing (aka SmartAd AB Data) [88] Single \- 2
/ 2 (B) Rand. Unit Overall result only Timestamp per RU Unknown Media & Ads
Display ads Unknown Jul 2020 Kaggle notebook Host: Kaggle DOI: N/A Licence: CC
BY-SA 3.0 Synthetical A/B-Tests [13] Multiple (25 856) \- 2 / 1 (R) Group.
Overall result only ✗ timestamp Synthetic N/A N/A N/A Kaggle notebook Host:
Kaggle DOI: Unknown Licence: CDLA-Sharing-1.0

Table 4.1: Results from the first ever survey of OCE datasets. The 13 datasets
identified are placed on the four taxonomy dimensions defined in Section 4.2,
together with the additional attributes recorded. In the Experiment Count
column, a second-line value (corresponding to Variant/Metric Count) of “$x$ /
$y$ (BCR)” means the dataset features $x$ variants and $y$ metrics in each
experiment, with the metrics based on Binary, Count, and Real-valued
responses. In the Response Granularity and Time Granularity columns,
Randomisation Unit is abbreviated as RU or Rand. Unit. Many resources are
accessed via links to non-scholarly articles – blog plots, Kaggle dataset
pages, and GitHub repositories. These resources may not persist over time.

###  4.4 Matching Dataset Taxonomy with Statistical Tests

Specifying the data requirements (or structure) and performing statistical
tests are perhaps two of the most common tasks carried out by data scientists.
However, the link between the two processes is seldom mapped out explicitly.
It is all too common to consider from scratch the question, “I need to run
this statistical test, how should I format my dataset?” (or more
controversially, “I have this dataset, what statistical tests can I run?”
[146]) for every new project/application, despite the list of possible dataset
dimensions and statistical tests remaining essentially the same.

We aim to speed up the process above by describing what summary statistics are
required to perform common statistical tests in digital experiments and link
the statistics back to the taxonomy dimensions defined in Section 4.2. The
exercise is similar to identifying the sufficient statistic(s) for a
statistical model [95]. However, the identification is done for the
encapsulating statistical inference procedure, with a practical focus on data
dimension requirements. We do so by stating the formula used to calculate the
corresponding effect sizes and test statistics and observing the summary
statistics required in common. The general approach also enables one to apply
the resultant mapping to any experiments that involve a two-sample statistical
test, including offline experiments and experiments without a randomised
control. We will refrain from discussing the full model assumptions and their
applicability for brevity. Instead, we point readers to the relevant work in
the literature.

####  4.4.1 Effect size and Welch’s $t$-test

We first look at the requirements to calculate effect sizes (using Cohen’s
$d$) and perform the most basic statistical test (using Welch’s
$t$-statistic). Recall from Sections 3.3.2 and 3.5.2 that the formulas to
calculate these two quantities are

$\displaystyle d$
$\displaystyle\,=\frac{\bar{Y}-\bar{X}}{\sqrt{\frac{(n-1)s^{2}_{X}+(m-1)s^{2}_{Y}}{n+m-2}}}\,,$
(*3.2) $\displaystyle T$
$\displaystyle\,=\frac{\bar{Y}-\bar{X}-\Delta}{\sqrt{\frac{s^{2}_{X}}{n}+\frac{s^{2}_{Y}}{m}}}\,,$
(*3.12)

where $(\bar{X},\bar{Y})$, $(s^{2}_{X},s^{2}_{Y})$, and $(n,m)$ are the means,
variances, and counts of the samples $X$ and $Y$, respectively.

We observe that we require only six _group-level_ summary statistics to
calculate the two quantities above: two means, two variances, and two counts.
Besides the counts, the exact list of quantities required depends on the
distributional assumptions used in a test, which dictates what the sufficient
statistics are.555Welch’s $t$-test assumes normally distributed data and hence
requires the sample mean and variance for both samples as they are a joint
sufficient statistic for normal distributions with unknown variance. For
decision metrics derived from binary responses, we only require the sample
mean as it is a sufficient statistic for a Bernoulli distribution. We call
these quantities _Dimension Zero (D0)_ quantities, as they are the bare
minimum required to run a statistical test. These quantities will be expanded
along the taxonomy dimensions defined in Section 4.2.

##### Cluster randomisation / dependent data

The sample variance estimates ($s^{2}_{X}$, $s^{2}_{Y}$) may be biased when
cluster randomisation is involved. Using the “free delivery” banner example
again, instead of randomly assigning each individual user to the control and
treatment groups, the business may randomly assign postcodes to the two
groups, with all users from the same postcode getting the same version of the
website. In this case, user responses may become correlated, violating the
independence assumptions in statistical tests. Common workarounds, including
the use of bootstrap [19] and the Delta method [64] (see Sections 5.4.1 and
5.5), generally require access to _sub-randomisation unit_ responses.

####  4.4.2 Experiments with adaptive stopping

As discussed in Section 4.1, experiments with adaptive stopping are getting
increasingly popular among the digital experimentation community. Here, we
motivate the data requirement for statistical tests in this domain by looking
at the quantities required to calculate the test statistics for a mixture
sequential probability ratio test (mSPRT, see Section 3.8 or [134]) and a
Bayesian hypothesis test using Bayes factor (see Section 3.9 or [66]), two
popular approaches in digital experimentation. Many other tests support
adaptive stopping [194, 271], though the data requirements should be largely
identical in terms of the dimensions defined in Section 4.2.

We first recall from Section 3.8 that running a mSPRT with a normal mixing
distribution ${H=\mathcal{N}(\theta_{0},\tau^{2})}$ involves calculating the
following test statistic upon observing the first $n$ samples from $X$ and
$Y$:

$\displaystyle\tilde{\Lambda}^{H,\theta_{0}}_{n}=\sqrt{\frac{\sigma^{2}_{X}+\sigma^{2}_{Y}}{\sigma^{2}_{X}+\sigma^{2}_{Y}+n\tau^{2}}}\exp{\left(\frac{n^{2}\tau^{2}(\bar{Y}_{n}-\bar{X}_{n}-\theta_{0})^{2}}{2(\sigma^{2}_{X}+\sigma^{2}_{Y})(\sigma^{2}_{X}+\sigma^{2}_{Y}+n\tau^{2})}\right)}\,,$
(*3.55)

where $\bar{X}_{n}=n^{-1}\sum_{i=1}^{n}X_{i}$ and
$\bar{Y}_{n}=n^{-1}\sum_{j=1}^{n}Y_{j}$ represent the sample mean of $X$ and
$Y$ up to sample $n$, respectively.

For Bayesian hypothesis tests, we recall from Section 3.9 that the test
statistic $\delta_{n,m}$ and effective sample size $E_{n,m}$ can be obtained
by decomposing the Wald test statistic upon observing the first $n$ samples
from $X$ and $m$ samples from $Y$:

$\displaystyle\sqrt{W_{n,m}}=\Bigg{(}\underbrace{\frac{\bar{Y}_{m}-\bar{X}_{n}}{\sqrt{\big{(}\frac{s^{2}_{X}}{n}+\frac{s^{2}_{Y}}{m}\big{)}/\left(\frac{1}{n}+\frac{1}{m}\right)}}}_{\triangleq\,\delta_{n,m}}-\underbrace{\frac{\Delta}{\sqrt{\big{(}\frac{s^{2}_{X}}{n}+\frac{s^{2}_{Y}}{m}\big{)}/\left(\frac{1}{n}+\frac{1}{m}\right)}}}_{\triangleq\,\Delta^{\\!\circ}}\Bigg{)}\underbrace{\frac{1}{\sqrt{\frac{1}{n}+\frac{1}{m}\vphantom{\big{(}\frac{s^{2}_{X}}{n}\big{)}}}}}_{\triangleq\sqrt{E_{n,m}}}\;,$
(*3.60)

where $\Delta^{\\!\circ}$ is the standardised effect size. $\delta_{n,m}$ and
$E_{n,m}$ are then used to calculate the Bayes factor under the hypothesis
pair $H_{0}:\Delta^{\\!\circ}=\theta_{0}$ and
${H_{1}:\Delta^{\\!\circ}\sim\mathcal{N}(\theta_{0},V^{2})}$:

$\displaystyle(BF_{10})_{n,m}=\frac{f(\delta_{n,m}\,|\,H_{1})}{f(\delta_{n,m}\,|\,H_{0})}=\frac{\phi\left(\delta_{n,m};\,\theta_{0},\,V^{2}+\frac{1}{E_{n,m}}\right)}{\phi\left(\delta_{n,m};\,\theta_{0},\,\frac{1}{E_{n,m}}\right)}\;,$
(*3.62)

where $\phi(\,\cdot\,;\mu,\sigma^{2})$ is the PDF of a normal distribution
with mean $\mu$ and variance $\sigma^{2}$.

During an experiment with adaptive stopping, we calculate the test statistics
stated above many times for different $n$ and $m$. This means a dataset can
only support the running of such experiments if it contains _intermediate
checkpoints_ for the counts ($n$, $m$) and the means
${(\bar{X}_{n},\bar{Y}_{m})}$, ideally _cumulative_ from the start of the
experiment. One often also requires the variances at the same time points (see
below). The only exception to the dimensional requirement above is where the
dataset contains responses at a _randomisation unit_ or finer level of
granularity and, despite recording the _overall results only_ , has a
_timestamp per randomisation unit_. Under this special case, we can still
construct the cumulative means ($\bar{X}_{n}$, $\bar{Y}_{m}$) for all relevant
values of $n$ and $m$ by ordering the randomisation units by their associated
timestamps.

##### Learning the effect size distribution (hyper)parameters

The two tests introduced above feature some hyperparameters ($\tau^{2}$ and
$V^{2}$) that must be specified or learned from data. These parameters
characterise the prior belief of the effect size distribution, which will be
the most effective if it “matches the distribution of true effects across the
experiments a user runs” [134]. Common parameter estimation procedures [1, 14,
115] require results from _multiple related experiments_.

##### Estimating the response variance

In the equations above, the response variance of the two samples,
$\sigma^{2}_{X}$ and $\sigma^{2}_{Y}$ (or $s^{2}_{X}$ and $s^{2}_{Y}$), are
assumed to be known before we observe all the samples. In practice, we often
use the plug-in empirical estimates $(s_{X}^{2})_{n}$ and $(s_{Y}^{2})_{m}$ –
the sample variances for the first $n$ samples from $X$ and $m$ samples from
$Y$, respectively. Thus, the data dimensional requirement is identical to that
of the counts and means as discussed above. We will also require a _sub-
randomisation unit_ response granularity when the plug-in estimates are biased
due to dependent data (see Section 4.4.1).

####  4.4.3 Non-parametric tests

We also briefly discuss the data requirements for non-parametric tests. Recall
from Section 3.7 that one of the most commonly used non-parametric tests in
digital experimentation, the Mann-Whitney $U$-test, calculates the following
test statistic:

$\displaystyle U=\sum_{i=1}^{n}\sum_{j=1}^{m}S(X_{i},Y_{j}),\quad\textrm{where
}S(X,Y)=\left\\{\begin{array}[]{ll}1&\textrm{if $Y<X$,}\\\ 1/2&\textrm{if
$Y=X$,}\\\ 0&\textrm{if $Y>X$.}\\\ \end{array}\right.$ (*3.49)

While a rank-based method is available for large $n$ and $m$, both methods
require knowledge of all the $X_{i}$ and $Y_{j}$. This requirement applies to
many other non-parametric tests (see Section 3.7). The observation suggests
that a dataset can only support a non-parametric test if it at least provides
responses at a _randomisation unit_ level.

We conclude by showing how we can combine the individual data requirements
above to obtain the requirement to design or run experiments for more
complicated statistical tests. This is possible due to the orthogonal design
of the taxonomy dimensions. Consider an experiment with adaptive stopping
using Bayesian non-parametric tests (e.g., with a Pólya Tree prior [43, 127]).
It involves a non-parametric test, requiring responses at a _randomisation
unit_ level. It computes multiple Bayes factors for adaptive stopping and
hence requires _intermediate checkpoints_ for the responses (or a timestamp
for each randomisation unit). Finally, it needs to learn the hyperparameters
of the Pólya Tree prior, which requires _multiple related experiments_. The
substantial data requirement along 3+ dimensions perhaps explains the lack of
relevant digital experiment datasets and the tendency for experimenters to use
simpler statistical tests for the day-to-day design or running of digital
experiments (or both).

###  4.5 A Novel Dataset for Experiments with Adaptive Stopping

We finally introduce the ASOS Digital Experiments Dataset, which we believe is
the first public dataset that supports the end-to-end design and running of
digital experiments with adaptive stopping. We motivate why this is the case,
provide a light description of the dataset (and a link to the more detailed
accompanying datasheet),2 and showcase the capabilities of the dataset via a
series of experiments. We also discuss the ethical implications of releasing
this dataset.

Recall from Section 4.4.2 that in order to support the end-to-end design and
running of experiments with adaptive stopping, we require a dataset that

  1. 1.

Includes _multiple_ related experiments;

  2. 2.

Is _real_ , so that any parameters learned are reflective of the real-world
scenario;

and either

  1. 3a.

Contains _intermediate checkpoints_ for the summary statistics during each
experiment (i.e., time-granular), or

  2. 3b.

Contains responses at a _randomisation unit_ granularity with a timestamp for
each randomisation unit (i.e. response-granular with timestamps).

None of the datasets surveyed in Section 4.3 meet all three criteria. While
the Upworthy [185], ASSISTments [244], and MeuTutor [262] datasets meet the
first two criteria, they all fail to meet the third.666All three report the
overall results only and hence are not time-granular. The Upworthy dataset
reports group-level statistics and hence is not response-granular. The
ASSISTments and MeuTutor datasets are response-granular but lack the timestamp
to order the samples. The Udacity Free Trial Screener Experiment dataset meets
the last two criteria by having results from a real experiment with daily
snapshots of the decision metrics (and hence time-granular), which supports
the running of an experiment with adaptive stopping. However, the dataset only
contains a single experiment, which does not help in learning the effect size
distribution (the design).

The ASOS Digital Experiments Dataset contains results from OCEs run by a
business unit within ASOS.com, a global online fashion retail company. In
terms of the taxonomy defined in Section 4.2, the dataset contains _multiple_
(78) _real_ experiments, with two to five variants in each experiment and four
decision metrics based on binary, count, and real-valued responses. The
results are aggregated on a _group_ level, with _daily or 12-hourly
checkpoints_ of the metric values _cumulative_ from the start of the
experiment. The dataset design meets all the three criteria stated above and
thus differentiates itself from other public datasets.

We provide readers with an accompanying datasheet (based on [99]) that
provides further information about the dataset. We also host the dataset on
Open Science Framework to ensure it is easily discoverable and can be
preserved long-term.2 It is worth noting that the dataset is released with the
intent to support development in the statistical methods required to run
digital experiments. The experiment results shown in the dataset are not
representative of ASOS.com’s overall business operations, product development,
or experimentation program operations, and no conclusion of such should be
drawn from this dataset.

####  4.5.1 Potential use cases

Figure 4.1: Distribution of $p$-values attained by the 99 OCEs in the ASOS
Digital Experiment Dataset using Welch’s $t$-tests, split by decision metrics.
The leftmost bar in each histogram represents experiments with $p<0.05$. Here,
we treat OCEs with multiple variants as multiple independent OCEs.

##### Meta-analyses

The multi-experiment nature of the dataset enables one to perform meta-
analyses. A simple example is characterising the distribution of $p$-values
(under Welch’s $t$-test) across all experiments (see Figure 4.1). We observe
that roughly a quarter of experiments in this dataset attain $p<0.05$ and
attribute this to the fact that what we experiment in digital experiments is
often guided by what domain experts think may have an impact. That said, we
invite external validation on whether there is evidence for data dredging
using, e.g., [193].

##### Design and running of experiments with adaptive stopping

We also demonstrate that the dataset can indeed support digital experiments
with adaptive stopping by performing a mixed sequential probability ratio test
(mSPRT) and a Bayesian hypothesis test via the Bayes factor for each
experiment and metric. This requires learning the hyperparameters $\tau^{2}$
and $V^{2}$. We learn, for each metric, a naïve estimate for $V^{2}$ by
collating the $\delta_{n,m}$ (see Equation (3.60)) at the end of each
experiment and taking their sample variance. This yields the estimates
1.30e-05, 1.07e-05, 6.49e-06, and 5.93e-06 for the four metrics respectively.

For $\tau^{2}$, we learn near-identical naïve estimates by collating the value
of Cohen’s $d$ (see Equation (3.2)) instead. However, as $\tau^{2}$ captures
the spread of unstandardised effect sizes, we specify in each test
$\tau^{2}=d\cdot(s^{2}_{X})_{n}$, where $(s^{2}_{X})_{n}$ is the sample
variance of all responses up to the $n^{\textrm{th}}$ observation in that
particular experiment. The Bayesian tests also require a prior belief in the
null hypothesis being true ($\mathbb{P}(H_{0})$) – we set it to $0.75$ based
on what we observed in the $t$-tests above.

We then calculate the $p$-value in mSPRT and the posterior belief in the
nullhypothesis ($\mathbb{P}(H_{0}|\textrm{data})$) in the Bayesian test for
each experiment and metric at each daily/12-hourly checkpoint, following the
procedures stated in [134] and [66, 143], respectively. We plot the results
for five experiments selected at random in Figure 4.2. It shows that the
$p$-value for an mSPRT is monotonically non-increasing. At the same time, the
posterior belief for a Bayesian test can fluctuate depending on the effect
size observed so far.

Figure 4.2: Change in (left) $p$-values in a mixed sequential probability
ratio test ($\tau^{2}=\texttt{5.92e-06}$) and (right) posterior belief in the
null hypothesis ($\mathbb{P}(H_{0}|\textrm{data})$) in a Bayesian hypothesis
test ($V^{2}=\texttt{5.93e-06}$, $\mathbb{P}(H_{0})=0.75$) during the
experiment for five experiments selected at random. The experiment duration
(x-axis) is normalised by its overall runtime. Only results for Metric 4 are
shown.

##### A quasi-benchmark for adaptive stopping methods

Real digital experiments, unlike machine learning tasks, generally do not have
a notion of ground truth. Nonetheless, the dataset can still be used as a
quasi-ground truth to compare two hyperparameter settings of the same adaptive
stopping method or between two adaptive stopping methods. For example, we can
treat the reject / not reject verdict from a Welch’s $t$-test at the end of an
experiment as the “ground truth” and compare it to the reject / not reject
verdict of an mSPRT at different experiment stages. This yields many
“confusion matrices” over different stages of individual experiments, where a
“Type I error” corresponds to cases in which a Welch’s $t$-test fails to
reject the null hypothesis and an mSRPT rejects the null hypothesis. A
confusion matrix for the end of each experiment can be seen in Table 4.2. As
the dataset was collected without early stopping, it allows us to perform
sensitivity analysis and optimisation on the hyperparameters of mSPRT under
what can be construed as a “precision-recall” trade-off of statistically
significant treatments.

##### Other use cases

The time series nature of this dataset enables one to detect bias (of the
estimator) across time, e.g., those caused by concept drift or feedback loops.
In the context of digital experiments, [41] described several methods to
detect invalid experiments over time that may be run on this dataset.
Moreover, with the time series spanning multiple experiments and decision
metrics, one can also learn the correlation structure across experiments,
decision metrics, and time [221, 274].

Table 4.2: Comparing the number of reject / not reject verdicts given by Welch’s $t$-test and an mSPRT at the end of an experiment for all four metrics. $t$-test | Rejects $H_{0}$ | Does not reject $H_{0}$
---|---|---
mSPRT | Rejects $H_{0}$ | Does not reject $H_{0}$ | Rejects $H_{0}$ | Does not reject $H_{0}$
Metric 1 | 19 | 7 | 2 | 71
Metric 2 | 20 | 7 | 18 | 49
Metric 3 | 16 | 9 | 6 | 63
Metric 4 | 16 | 11 | 4 | 63

####  4.5.2 Ethical considerations

We finally discuss the ethical implications of releasing the dataset, touching
on data protection and anonymisation, potential misuse, and the ethical
considerations for running digital experiments in general.

##### Data protection and anonymisation

The dataset records aggregated activities of hundreds of thousands or millions
of website users for business measurement purposes. Hence, it is impossible to
identify a particular user. Moreover, to minimise the risk of disclosing
business sensitive information, all experiment context is either removed or
anonymised. One should not be able to tell who is in an experiment, when the
experiment is run, what treatment the experiment involves, and what decision
metrics are used. We refer readers to the accompanying datasheet2 for further
details in this area.

##### Potential misuses

A digital experiment dataset, no matter how anonymised it is, reflects the
behaviour of its participants under a specific application domain and time. We
urge potential users of this dataset to exercise caution when attempting to
generalise the learnings. It is important to emphasise that the learnings
differ from the statistical methods and processes demonstrated on this
dataset. We believe the latter are generalisable, i.e., they can be applied to
other datasets with similar data dimensions regardless of the datasets’
application domain, target demographics, and temporal coverage, and appeal for
potential dataset users to focus on such.

One example of generalising the learnings is using this dataset as a full
performance benchmark. As discussed above, this dataset does not have a notion
of ground truth, and any quasi-ground truths constructed are themselves a
source of bias to estimators. Thus, experiment design comparisons need to be
considered at a theoretical level [176]. Another example will be directly
applying the value of hyperparameter(s) obtained while training a model on
this dataset to another dataset. While this may work for similar application
domains, the less similar they are, the less likely the hyperparameters
learned will transfer. This may risk introducing bias, both in the estimator
and in fairness.

##### Running digital experiments in general

The dataset is released to support experiments with adaptive stopping, which
will enable a faster experimentation cycle. The ethical concerns will
naturally mount as we run more digital experiments, which are ultimately human
subjects research. We reiterate the importance of the following three
principles when we design and run experiments [206, 208]: respect for persons,
beneficence (properly assess and balance the risks and benefits), and justice
(ensure participants are not exploited), and refer readers to Chapter 9 of
[155] and its references for further discussions in this area.

###  4.6 A Brief Recap

Digital experiments are a powerful tool for online organisations to assess the
impact of their digital products and services. To safeguard future
methodological development in the area, it is vital that we have access to and
a systematic understanding of relevant datasets arising from real experiments.
We described the result of the first ever survey on publicly available OCE
datasets. We also provided a dimensional taxonomy that links the data
collection and statistical test requirements for digital experiments in
general. In addition, we released the first ever dataset that can support
digital experiments with adaptive stopping, which design is grounded on a
theoretical discussion between the taxonomy and statistical tests. Via
extensive experiments, we also showed that the dataset is capable of
addressing the identified gap in the literature.

Our work on surveying, categorising, and enriching the publicly available
digital experiment datasets is just the beginning, and we invite the community
to join in the effort. As discussed above, we have yet to see a dataset that
can support methods dealing with correlated data due to cluster randomisation,
or the end-to-end design and running of experiments with adaptive stopping
using Bayesian non-parametric tests. We also see ample opportunity to
generalise the survey to cover datasets arising from uplift modelling tasks,
quasi-experiments, and observational studies. Finally, we can further expand
the taxonomy, which already supports datasets from all experiments, with extra
dimensions (e.g., number of features to support stratification, control
variate, and uplift modelling methods) as the area matures.

##  Chapter 5 Recent Advances in Digital Experimentation Methods

Part of the chapter (Section 5.5) is adapted from the research paper
“Measuring e-Commerce Metric Changes in Online Experiments”, presented at ACM
Web Conference 2023 (WWW ’23) [177].

###  5.1 Motivation

After establishing the statistical tests and datasets required to run a
digital experiment in Chapters 3 and 4, we can now describe what a digital
experiment entails. To begin, we outline the causal reasoning behind
experiments. Establishing causation (i.e., making conclusions such as “this
treatment causes that business metric to change”) is a step further than
establishing correlation111Also known as _association_ within the causal
inference community [122]. that a statistical test may do. We thus require
additional theoretical grounding to represent the concept of causation. This
chapter provides an initial view of the potential outcomes framework / Rubin
causal model and its associated language.222Other tools and frameworks exist,
e.g., directed acyclic graphs and do-calculus, which are arguably more
generalisable. Here, we prioritise the ease of understanding.

Using the potential outcomes framework, we can describe different classes of
experiments. The classes, primarily characterised by the level of control an
experimenter has, are randomised controlled trials (RCTs), quasi-experiments,
and natural experiments. Having a classification of digital experiments
enables us to highlight the different methodological and implementation
challenges faced by digital experimenters on each front.

Given the availability of excellent and up-to-date review articles that deal
with established and emerging approaches in each area, it is perhaps not the
most effective for us to cover yet another systemic review of digital
experimentation methods. Thus, we provide a more lightweight, narrative
overview beyond introducing the review articles. We will describe the major
challenges and highlight several prominent works for each challenge as entry
points for readers interested in further exploration.

As we continue to place equal emphasis on the theory and practice of digital
experimentation methods, we also enrich the overview with case studies. These
case studies provide spotlights on digital randomised controlled trials with
dependent responses and quasi-experiments that assign treatments based on
geographical regions. In addition to providing evidence on the extent of these
challenges, the case studies also detail the practical considerations when
implementing the methods to address them.

The rest of the chapter is organised as follows. We introduce the basics of
the potential outcomes framework in Section 5.2. Using its notation and rules,
we outline the three classes of experiments in Section 5.3. We then interleave
the reviews on advances in each class of experiments (RCTs in Section 5.4 and
quasi-experiments/natural experiments in Section 5.6) with spotlights on
methods that deal with dependent responses (Section 5.5) and geo-experiments
(Section 5.6.3).

###  5.2 From Correlation to Causation – The Potential Outcomes Framework

Readers may recall that in Chapter 3, we noted that statistical tests alone
are insufficient to establish causal relationships between treatments and
changes in responses. In this section, we bridge the gap between
correlation/association thinking (as featured in statistical tests) and causal
thinking (as featured in experiments and causal inference) by introducing the
most basic concepts of the potential outcomes framework. In subsequent
sections, we will refer to and extend these concepts.

This introduction extends the notations in Chapter 3 and takes a more
leisurely pace than other introductions to background materials in the thesis.
This is deliberate – in the author’s experience, it is a big jump to fully
appreciate the framework’s premises, rules, and notations, especially for
those who are used to dealing only with randomised control trials. Readers
familiar with the potential outcomes framework and the $Y_{i}(0)$-style
notation in causal inference may consider skimming through the section. They
may also find other expositions on the framework [62, 122, 131] useful.

##### Observed responses

In Chapter 3, we often deal with two sets of responses: $X_{1},\cdots,X_{n}$
and $Y_{1},\cdots,Y_{m}$. In the general context of statistical testing, the
responses can arise from two arbitrary sources, e.g., the height of males vs
females in a survey or ice cream sales vs the number of drowning incidents
[201]. In the context of an experiment, the two sets of responses would have
come from experimental units that are assigned and subjected to two different
treatments.333We focus on the two-treatment case for the rest of the chapter.
The numerical designation of the treatment assignments, i.e., 0, 1, $\cdots$,
eases the extension of the framework to cover multiple treatments. Formally,
by modelling the treatment assignment using a binary r.v. $\mathcal{A}$ and
denoting the two possible assignments $\mathcal{A}=0$ and $\mathcal{A}=1$, we
can represent the responses under the additional context as

$\displaystyle X_{i}\,|\,\mathcal{A}=0\quad\textrm{ and }\quad
Y_{j}\,|\,\mathcal{A}=1\,.$ (5.1)

Assuming all $X_{i}$ and $Y_{j}$ are identically distributed copies of their
respective populations, by taking the sample mean of the two sets of
responses, we obtain an unbiased estimator of the expected response of the
overall populations $X$ and $Y$ given the corresponding treatments:

$\displaystyle\mathbb{E}(X\,|\,\mathcal{A}=0)\quad\textrm{ and
}\quad\mathbb{E}(Y\,|\,\mathcal{A}=1)\,.$ (5.2)

##### Counterfactual responses

Note $X_{i}$ and $Y_{j}$ in Expression (5.1) are merely the _observable_
responses from their associated experimental units. We do not observe how the
experimental unit responding $X_{i}$ under ${\mathcal{A}=0}$ would have
responded had ${\mathcal{A}=1}$, nor do we observe how the unit responding
$Y_{j}$ under $\mathcal{A}=1$ would have responded had $\mathcal{A}=0$. These
“would haves” are known as the _counterfactual_ responses and are essential to
causal reasoning – understanding the impact of a treatment requires
understanding both the outcomes when the treatment is and is not prescribed.

The potential outcome framework unifies the observed and counterfactual
responses from the same experimental unit by treating all equally as potential
responses (or _outcomes_) and giving each potential response a separate
designation. In the two-treatment case, they are denoted as

$\displaystyle X_{i}\triangleq\begin{cases}X_{i}(0)&\textrm{ if
}\mathcal{A}=0\\\ X_{i}(1)&\textrm{ if }\mathcal{A}=1\,,\end{cases}\qquad
Y_{j}\triangleq\begin{cases}Y_{j}(0)&\textrm{ if }\mathcal{A}=0\\\
Y_{j}(1)&\textrm{ if }\mathcal{A}=1\,.\end{cases}$ (5.3)

More generally, we can write $X_{i}=X_{i}(a)$ if $\mathcal{A}=a$ and
$Y_{j}=Y_{j}(a)$ if $\mathcal{A}=a$, where ${a\in\\{0,1\\}}$.444In [122], the
same quantities are denoted $X_{i}^{a}$ and $Y_{j}^{a}$, sometimes with the
numeric value of $a$ explicitly specified, e.g., $Y_{j}^{a=0}$ for
disambiguation. The framework features a missing data problem – some potential
responses (generally one max per experimental unit) are observable, and others
must be estimated.

Using the framework, we can define the causal impact of a treatment over
another more formally. In a two-treatment setting, the treatment effect (of
treatment 1 over 0) on a single experimental unit that responds $X_{i}$ or
$Y_{j}$ is

$\displaystyle\Delta_{X_{i}}=X_{i}(1)-X_{i}(0),\quad\Delta_{Y_{j}}=Y_{j}(1)-Y_{j}(0)\,.$
(5.4)

Assuming again that all $X_{i}$ and $Y_{j}$ are identically distributed copies
of their respective populations, we can also define the average treatment
effect (ATE) across the overall populations $X$ and $Y$ as

$\displaystyle\Delta_{X}=\mathbb{E}(X(1)-X(0)),\quad\Delta_{Y}=\mathbb{E}(Y(1)-Y(0))\,.$
(5.5)

The framework also enables us to formalise the adage “correlation does not
imply causation”. So far, we have encountered the expected observed response
given a treatment $\mathbb{E}(X\,|\,\mathcal{A}=a)$ (see Expression (5.2)) and
the expected potential response under the said treatment $\mathbb{E}(X(a))$.
They may appear to be descriptions of the same quantity but are not the same.
This is because a third variable (e.g., a confounder) may affect both $X$ and
$\mathcal{A}$.

To illustrate, we use the “ice cream consumption causes drowning” fallacy
[201] – let $X$ be the number of (observed) drowning incidents and
$\mathcal{A}$ be a binary r.v., representing high ($\mathcal{A}=1$) and low
($\mathcal{A}=0$) ice cream sales. It is established that the number of
drowning incidents increases along with ice cream sales. In other words,
$\mathbb{E}(X\,|\,\mathcal{A}=1)$ is high, $\mathbb{E}(X\,|\,\mathcal{A}=0)$
is low, and the difference
$\mathbb{E}(X\,|\,\mathcal{A}=1)-\mathbb{E}(X\,|\,\mathcal{A}=0)$ is positive.
However, it is also established that ice cream consumption does not cause
drowning, nor does drowning incidents cause increased ice cream sales.555It is
conceivable for drowning incidents to cause _decreased_ ice cream sales due to
emergency responses. Instead, the correlation/association in the expected
observed responses arises due to a third factor: weather. Hot and sunny spells
generally raise the demand for ice cream and, unfortunately, the number of
drowning incidents as more people visit various water bodies and thus are
exposed to the risk of drowning.

The expected potential responses better reflect the lack of a causal
relationship between ice cream sales and drowning incidents. Given the
potential number of drowning incidents under high ice cream sales
$\mathbb{E}(X(1))$, one would expect the counterfactual response
$\mathbb{E}(X(0))$, i.e., the number of drowning incidents had the ice cream
sales been low _and all else equal_ , will be similar if not the same. This
leads to a negligible causal impact $\mathbb{E}(X(1))-\mathbb{E}(X(0))$.

###  5.3 Classes and Examples of Digital Experiments

Figure 5.1: The onion model for different types of experiments, in decreasing
degree of control from the experimenter (inner to outer): Randomised
controlled trials, quasi-experiments, and natural experiments. Quasi-
experiments and randomised controlled trials are differentiated by the
experimenter’s ability to assign experiment participants randomly into
different groups for different treatments. Natural experiments and quasi-
experiments are differentiated by the experimenter’s ability to vary the
treatment (or lack thereof) prescribed to an experiment participant.

To aid the discussion in the rest of the chapter, we provide an overview of
different classes of experiments defined in the traditional statistics
literature and further examples within digital experimentation. Figure 5.1
shows the classes of experiments and what differentiates a class from another.

####  5.3.1 Randomised controlled trials

The class of experiments where experimenters have the most control is
randomised controlled trials (RCTs). They are also known as online controlled
experiments (OCEs) in the digital experimentation community [155]. The free
delivery banner experiment described at the beginning of the thesis (see
Figure 1.1) belongs to this class. These experiments are characterised by the
experimenter’s ability to apply a scientific control and randomly assign
experimental units to different groups.666Unlike in medical trials, the issue
of _blinding_ , where information that may influence the result is withheld
from the experimenter or experiment participants (or both) to minimise biases,
is rarely discussed in digital experimentation. It is often addressed by
having the Internet between the experimenter and participants: the
overwhelming majority of participants are unaware of what treatment(s) are in
place when they visit a website. Moreover, experimenters would only know who
the participants are during post-experiment analysis as they delegate the task
of assigning and tracking participants to automated computer systems. For the
free delivery banner example, this corresponds to the ability to show a
version of the website without the “free delivery” banner and randomly assign
users to see different website versions, respectively.

Most of the value of an RCT comes from the ability to perform random
assignments, which has two major implications. Here, we describe the
implications using the potential outcomes framework. Firstly, randomisation
generally implies exchangeability.777Also known as _unconfoundedness_ or
_ignorability_. While there are randomisation schemes that lead to non-
exchangeable potential responses, in practice, almost all online controlled
experiments rely on randomisation schemes (e.g., simple randomisation) that
yield some form of exchangeability [240]. This means the potential responses
and the actual treatment are independent. We denote such exchangeability as
$X(0),X(1)\perp\kern-6.00006pt\perp\mathcal{A}$ and
$Y(0),Y(1)\perp\kern-6.00006pt\perp\mathcal{A}$ and observe that if the
potential responses are exchangeable,

$\displaystyle\mathbb{E}(X(a))$
$\displaystyle=\mathbb{E}(X(a)\,|\,\mathcal{A}=0)=\mathbb{E}(X(a)\,|\,\mathcal{A}=1)\,\textrm{
and }$ $\displaystyle\mathbb{E}(Y(a))$
$\displaystyle=\mathbb{E}(Y(a)\,|\,\mathcal{A}=0)=\mathbb{E}(Y(a)\,|\,\mathcal{A}=1)\,,\;a\in\\{0,1\\}\,.$
(5.6)

Secondly, as experimental units in both groups can be viewed as a simple
sample of the same underlying population under randomisation, the r.v.’s
representing the potential responses of _both_ groups under the same
treatment, i.e.,

$\displaystyle X_{1}(a),\cdots,X_{n}(a),Y_{1}(a),\cdots,Y_{m}(a)\textrm{ for
}a\in\\{0,1\\}\,,$ (5.7)

are interchangeable.888Also known as an _exchangeable sequence of r.v.’s_
[44]. There are a few subtle differences between the two exchangeability
definitions. Thus, we use a different name to disambiguate these two concepts.
This means they can be swapped within and across the groups without
consequences.

The two implications mean we can obtain an unbiased estimate/estimator on the
average treatment effect using only the observed responses under
randomisation. To illustrate, we can express the ATE defined in Equation (5.5)
as

$\displaystyle\Delta_{X}\,$ $\displaystyle=\mathbb{E}(X(1)-X(0))$ (By
definition – Eq. (5.5)) $\displaystyle=\mathbb{E}(X(1))-\mathbb{E}(X(0))$
(Linearity of expectation) $\displaystyle=\mathbb{E}(Y(1))-\mathbb{E}(X(0))$
(Interchangeability between $X(1)$ and $Y(1)$)
$\displaystyle=\mathbb{E}(Y(1)\,|\,\mathcal{A}=1)-\mathbb{E}(X(0)\,|\,\mathcal{A}=0)$
(Exchangeability of $Y(1)$ and $X(0)$ – Eq. (5.6))
$\displaystyle=\mathbb{E}(Y\,|\,\mathcal{A}=1)-\mathbb{E}(X\,|\,\mathcal{A}=0)$
$\displaystyle\textrm{(By definition -- Eq.\leavevmode\nobreak\
\eqref{eq:rade_pof_counterfactual_response})}\,.$ (5.8)

We can also show that $\Delta_{Y}$ is equal to the last line of Equation (5.8)
using the same arguments (plus the interchangeability between $X(0)$ and
$Y(0)$). This should not come as a surprise as $X$ and $Y$ are based on
experimental units from the same population and thus should share the same
ATE. As stated in Section 5.2, estimating the last line of Equation (5.8) is
simply a matter of taking two sample means over the two groups of observed
responses. This has the effect of reducing the ATE estimation problem into a
much more straightforward two-sample testing problem. Experimenters can use
the many readily available methods in statistical testing (see Chapter 3) and
avoid estimating expected counterfactual responses, which may attract
considerable bias.

To sum up, the results above show that randomisation generally implies that
any difference in the responses can be attributed to the causal effect of the
treatment and not to other measured or unmeasured confounding. Such a
desirable property perhaps explains why RCTs are considered the “gold
standard” in medical trials [139] and are the most common type of digital
experiment. We will outline further statistical challenges to running RCTs
online in Section 5.4.

####  5.3.2 Quasi-experiments

Taking away the ability to randomly assign experiment units in randomised
controlled trials leaves us with a quasi-experiment. Consider the example in
[287], where a digital organisation developed a mobile application
downloadable from app stores such as Apple App Store and Google Play. The
organisation is interested in whether a new app feature shipped within a new
app version will improve a business metric relative to the existing app
version. For the new feature to reach app users, developers usually upload the
new version onto the app stores. The app stores then have to review the new
app version, and once approved, app users have to update their app.

In this scenario, the ability to assign who sees the new or existing version
of the app (let alone randomly) is taken away from the experimenters and put
in the hands of the users, who decide when to update the app, and the app
stores, who decide when to allow the new version of the app to be downloaded.

The inability to randomise treatment assignments generally marks the beginning
of causal inference’s remit. Experimenters can no longer expect the
exchangeability mentioned in the previous subsection to automatically hold and
have to employ other methods to estimate the counterfactual responses from one
or more observed responses. In the language of the potential outcomes
framework, we want to estimate $X_{i}(1)$ and $Y_{j}(0)$ using
$X_{i}\,|\,\mathcal{A}=0$ and $Y_{j}\,|\,\mathcal{A}=1$, where $X_{i}$
represents the responses from users who have yet to download the new app
version, $Y_{j}$ represents the responses from users who have seen the new
version (and presumably interacted with the new feature), and $\mathcal{A}$ be
the binary treatment assignment indicator. We will introduce several causal
inference approaches and how they apply to quasi-experiments in Section 5.6.

####  5.3.3 Natural experiments

If we further remove the ability to apply scientific control,999Also known as
_manipulating_ the treatment or independent variables [231]. we end up with
natural experiments. While experimenters cannot apply different treatments (in
this case, different versions of a digital product or experience),
participants may still encounter scenarios that include one or more of the
said treatments by the act of nature. This means it remains possible to draw
inferences on the causal effect of the treatments from observational data.

An experimenter may opt not to apply a scientific control and run a natural
experiment instead due to various operational, reputational, legal and ethical
considerations. For example, experiments aiming to measure the value added by
existing call centre support may be prevented from actively blocking some
users’ access to the call centre as it negatively impacts user experience.
Other experiments seeking to understand the price elasticity of
demand101010The relationship between the price of a product/service sold by an
organisation and its demand. may be keen to avoid engaging in the
controversial practice of price discrimination111111The act of offering
different users a different price for the same digital product/service. to
avoid negative press and risk of legal actions [73, 282]. Lastly, testing
different news headlines simultaneously in front of readers may also damage
editorial credibility or even “fan the flames” of partisan political beliefs
in the long run [133]. Experimenters may instead leverage responses from users
who have never contacted the call centre, sales on similar digital
products/services priced differently, and reactions on past news headlines on
similar topics to understand the respective causal impact.

Most natural experiments share the same causal inference approaches as quasi-
experiments. We will cover a few applications in Section 5.6.

####  5.3.4 Special case – Experiments for personalisation strategies

It is worth noting that the boundaries between each experiment class
introduced above are not clear-cut. Depending on the situation, experimenters
may have various degrees of control over random assignments.

Consider a follow-up experiment to our free delivery banner experiment
introduced above. Suppose the organisation is now interested in whether
varying the _timing_ of showing the “free delivery” banner will lead to
further improvement. One can run an experiment as simple as comparing “showing
the banner after a user has viewed _three_ shop products” vs “showing the
banner after a user has viewed _four_ shop products” and see which timing rule
leads to a better outcome.

In the experiment, the treatment (timing rule) a user is subjected to is
_jointly_ determined by the assignment from experimenters and the user’s
actions. While the experimenters can randomly assign which timing rule to put
the user through, the user must also qualify under the rules by seeing three
or four shop products. Only by doing so, the user would see the “free
delivery” banner, without which the notion of treatment effect would not make
sense. This means experimenters have some, but not complete, control in random
assignments – more than that in the new app feature experiment described in
Section 5.3.2, but certainly less than the vanilla free delivery banner
experiment discussed in Section 5.3.1.121212Some literature describes this
scenario as _one-sided non-compliance_ [270].

The banner display timing experiment is one of many experiments for
_personalisation strategies_ – complex sets of targeted user interactions
common in e-commerce and digital marketing.131313The example provided is
grossly simplified to facilitate discussion. In practice, display timing
strategies, in true personalisation spirit, usually specify how different
users are shown their “free delivery” banner at a different stage of their
website visit, depending on their browsing behaviour. Thus, an experiment that
compares two display timing strategies is essentially comparing the execution
of two _sets_ of display timing rules generated from two meta-principles. We
will focus on the design of these experiments and further elaborate on their
unique challenges in Chapter 6.

###  5.4 Advances in Online Controlled Experiments

Here, we provide a brief review of the advances in OCEs. Since its inception
in the late 1990s [153], the field has seen tremendous growth in research
output and its economic and societal impact [263]. More recently, we have also
seen some comprehensive and up-to-date reviews in the area:

  * •

[155] provided a narrative, general-purpose guide to the field while collating
many relevant works.141414The author reviewed part of the manuscript before
the book’s publication in 2020.

  * •

[12] and [225] provided systematic literature reviews. The former built a
taxonomy based on the process-infrastructure distinction, type of
methodological approach, and the challenges/benefits involved. The latter
categorised the works by the subject of an experiment,151515Defined in [225]
as the application domain and the element subjected to experimental
manipulation. choices made in the experiment design/execution processes, and
the role of stakeholders.

  * •

[116] and [159] described the challenges. The former provided a general view
by including data, engineering, and organisational culture topics. The latter
focused on the statistical aspect and provided a more technical treatment to
each topic presented.

This review aims to be a gateway to the multitude of research works in the
area by unifying the many nomenclatures, providing a concise problem statement
for each challenge, and suggesting works accessible to those new to a
particular line of work. For a fuller exposition of the challenges and
research in this area, readers should refer to the five high-quality reviews
listed above.

Most of the section is dedicated to advanced statistical methods, building on
concepts and notations covered in Chapter 3 (on statistical testing) and
Sections 5.2 and 5.3.1 (on the potential outcomes framework). We will also
briefly outline works addressing other topics in running successful OCEs, as
mentioned in the second research question in Section 1.2. These include
choosing decision metrics / evaluation criteria, scalable computer systems,
and fostering a growth mindset organisational culture.

####  5.4.1 Advanced statistical methods

We begin with the statistical challenges in OCEs that necessitate the
development of advanced methods. These challenges included in the section are
the multiple testing/comparison problem, post-selection inference,
heterogeneous treatment effects, dependent responses, stable unit treatment
value assumption (SUTVA) violations, and improving experiment sensitivity.

##### Multiple testing/comparisons

Recall from Section 3.6.3 that the probability of rejecting the null $H_{0}$
in an NHST when $H_{0}$ is true (i.e. the false positive rate) is $\alpha$.
The false positive rate is usually small so that when a test rejects $H_{0}$,
we have reasonable conviction to hail the experiment result as a discovery.
That said, the probability of committing a false positive accumulates as we
perform additional comparisons. For $N_{\textrm{test}}$ independent NHSTs, we
expect $(\alpha\cdot N_{\textrm{test}})$ tests to reject $H_{0}$ when $H_{0}$
is true. This means that for every 20 NHSTs run under $\alpha=0.05$, we expect
the tests to reject $H_{0}$ once on average, even when all the $H_{0}$ are
true [202]. Hailing all test rejections as discoveries without accounting for
the inflated false positive rate will lead to biased decisions.161616This is
closely related to the continuous monitoring/peeking problem discussed in
Section 3.8 and Section 4.1. That said, comparisons in the continuous
monitoring/peeking problem are often correlated, leading to the risk of
committing a false positive accumulating differently.

The problem is increasingly prevalent due to the increasing number of
experiments, treatment pairs, and user segments to compare (see paragraph on
heterogeneous treatment effect below). Usually, experimenters seek to control
the family-wise error rate (FWER) or the false discovery rate (FDR), defined
over many experiments as

FWER $\displaystyle\triangleq\mathbb{P}(\textrm{\\# Type I error}\geq
1)=1-\mathbb{P}(\textrm{\\# Type I error}=0),$ (5.9) FDR
$\displaystyle\triangleq\mathbb{E}\left(\frac{\textrm{\\# Type I
error}}{\textrm{\\# Test rejecting }H_{0}}\right)\,.$ (5.10)

The most common techniques that control for FWER include Bonferonni correction
[80] and Šidák correction [249], whereas those for FDR include the Benjamini-
Hochberg [25] and knockoff procedures [97].

##### Post-selection inference

Having multiple comparisons over experiments, treatment pairs and sub-
populations will remain a potential source of bias even when we move away from
binary decisions and on to effect size estimation. Consider a two-sample,
right-tailed $t$-test (and all its associated notation, see Section 3.5.2). In
addition to deciding to reject/not reject $H_{0}:\Delta=\theta_{0}$, we also
estimate the treatment effect $\Delta$. We know $\bar{Y}-\bar{X}$ is an
unbiased estimator for $\Delta$. However, with many statistical tests, we
generally ignore all tests that fail to reject $H_{0}$ – assuming
$\Delta=\theta_{0}$ as per the null hypothesis – and focus on tests that
reject $H_{0}$. The treatment effect estimate after such test selection now
overestimates $\Delta$:

$\displaystyle\mathbb{E}\left(\bar{Y}-\bar{X}\,|\,T>t_{\nu,1-\alpha}\right)=\mathbb{E}\left(\bar{Y}-\bar{X}\,\bigg{|}\,\bar{Y}-\bar{X}>\theta_{0}+t_{\nu,1-\alpha}\sqrt{\frac{s^{2}_{X}}{n}+\frac{s^{2}_{Y}}{m}}\right)>\mathbb{E}\left(\bar{Y}-\bar{X}\right)=\Delta\,.$
(5.11)

Another way to put this in the context of multiple comparisons is that the
test selection process above favours “lucky” OCEs, some of which are merely
false discoveries. Inferring quantities with the test selection process is an
example of _post-selection inference_ [65], and the estimation bias it
introduces is also known as the _winner’s curse_ [161].

The best way to detect the existence of the winner’s curse and correct the
bias is via replication [65]. In practice, it is often operationally
challenging. It also seems unnecessary if one only seeks the aggregated
treatment effect from multiple successful OCEs; in this case one can employ a
maximum likelihood approach to estimate the expected bias from the test
selection process [161]. To estimate the treatment effects of individual OCEs,
one can perform pseudo-replication by splitting an experiment into two sub-
experiments with the same treatments and build a regression model on the
expected treatment effect of the second sub-experiment (without any selection)
using that of the first sub-experiment (which goes through test selection)
[45]. They can also use empirical Bayes methods [14] such as Tweedie’s formula
and the James–Stein estimator [85]. Finally, [65] combined elements of the
approaches above to improve the precision of individual treatment effect
estimates.

##### Heterogeneous treatment effects

Many experiments seek to measure the average treatment effect across the
population. That said, it is valuable for experimenters to understand whether
there are subgroups whose treatment effect may deviate from that of the entire
population [79]. To detect such heterogeneous treatment effects, experimenters
often estimate the conditional average treatment effect (CATE) instead, i.e.

$\displaystyle\mathbb{E}(Y_{j}(1)-Y_{j}(0)\,|\,\mathcal{C}_{j}),$ (5.12)

where $\mathcal{C}_{j}$ represents the covariates for experimental unit $j$.
Many works assume conditional interchangeability and exchangeability, such
that they can estimate CATE using the conditional observed responses
$\mathbb{E}(X_{i}\,|\,\mathcal{C}_{i}=c)$ and
$\mathbb{E}(Y_{j}\,|\,\mathcal{C}_{j}=c)$ from an OCE. See Section 5.6.1 for
further discussions on CATE estimation and Section 3 of [159] for a summary of
works in the area.

##### Dependent responses

Many OCEs feature statistical tests (see Section 3.5) that assume i.i.d.
responses. While many OCEs do have observations that are approximately i.i.d.
[67], such assumptions are not always justified. We have (implicitly) covered
deviations from the identically distributed assumptions when we discussed
heterogeneous treatment effects above. Below, we look at deviations from the
independence assumptions.

One can find dependent responses in many experiment designs. A significant
portion features dependent treatment assignments. This could be explicitly
imposed by randomly assigning pre-existing groups of experimental units
(instead of individuals) to treatment groups, also known as _cluster
randomisation_ , as briefly discussed in Section 4.4.1. It could also
implicitly arise as one randomises individual experimental units independently
yet observes and infers based on more granular responses, effectively forming
a cluster per experimental unit.171717Digital experimenters often
differentiate an experimental unit, a _randomisation unit_ (one subjected to
independent randomisation), and an analysis unit (one that yields a response).
These units do not necessarily coincide [70, 67]. In Section 5.5, we will
present an extensive case study on dependent responses in e-commerce OCEs,
which arises from experimenters randomising by users and measuring changes to
business metrics based on more granular transaction- or item-based responses.

Having dependent responses can affect treatment effect estimates and decisions
from statistical tests. Consider a one-sample $t$-test (see Section 3.5.2 for
all its associated notation). We recall from Equation (3.11) that its test
statistic is $(\bar{X}-\mu_{X})/\sqrt{s_{X}^{2}/n}$. Moreover, the test
statistic’s denominator is an estimator of the standard error of the mean
(SE):

$\displaystyle\sqrt{\textrm{Var}(\bar{X})}$
$\displaystyle=\sqrt{\frac{1}{n^{2}}\big{(}\textstyle\sum_{i=1}^{n}\textrm{Var}(X_{i})+2\textstyle\sum_{i<j}\textrm{Cov}(X_{i},X_{j})\big{)}}\;.$
(5.13)

Under i.i.d. assumptions, $\textrm{Cov}(X_{i},X_{j})=0\;\;\forall i\neq j$ and
thus
${\textrm{Var}(\bar{X})=\sum_{i}\textrm{Var}(X_{i})/n^{2}=\sigma^{2}_{X}/n}$.
This implies $\sqrt{s_{X}^{2}/n}$ is an unbiased estimator of the SE. However,
if $X_{i}$ are dependent, then $\textrm{Cov}(X_{i},X_{j})\neq 0$, which makes
$\sqrt{s_{X}^{2}/n}$ a biased estimator. In the case study below, we will
demonstrate how the bias affects the power and confidence interval
calculations and, thus, the decisions made in statistical tests and
experiments.

Digital experimenters use the delta method [64] or one-way/block bootstrap
[19] to mitigate the bias. We will provide a critique of these methods in the
case study below. Another interesting work in the area is [67], which explores
scenarios where an experimenter can treat responses as i.i.d. before providing
a variance formula that exploits the relationship between the
experimental/randomisation/analysis units and accommodates a wide range of
randomisation mechanisms in practical settings.

##### Stable unit treatment value assumption (SUTVA) violations

Responses can also become dependent by violating the _stable unit treatment
value assumption_ (SUTVA), an assumption we made implicitly when we defined
the potential outcomes in Section 5.2 / Equation (5.3).181818Also known as
having _spillover effects_ [250, 253], in the sense that a treatment spills
over to experimental units in another treatment group (due to, e.g.,
interactions between experimental units in different groups) and induces
unwanted treatment effects. More precisely, SUTVA is an assumption in two
parts – in the words of [131]:

  1. 1.

(No interference [51, 234]) – “The potential outcomes for any [experimental]
unit do not vary with the treatments assigned to other units, and,”

  2. 2.

(No hidden variations of treatments) – “[F]or each [experimental] unit, there
are no different forms or versions of each treatment level, which lead to
different potential outcomes.”

In OCEs, interference mainly arises due to the network effects. As
experimental units interact with each other in the network, their potential
response no longer depends on what treatment they have received but also on
what treatments their connections have received. In other words, a treatment
may spill over to an experimental unit’s connections. The effect can be found
directly in social networks (where a user interacts with their connections)
[142] or indirectly via marketplaces (where suppliers/buyers compete for
shared resources) [167] and recommender systems (where the system recommend
multiple items/users to the same user/item) [221].

While dependent treatment assignment, which breaks the independence assumption
but maintains SUTVA, generally affects the variance estimate, a SUTVA
violation can bias the treatment effect estimates. [19] have shown that
interaction amongst users and, separately, the items recommended can result in
an increased false positive rate in experiments when applied to Facebook data,
likely due to bias in both the treatment effect and variance estimates.

A common way to mitigate the spillover effect in social networks is to
randomise treatment group assignments over clusters191919In network speak,
sub-graphs that are highly connected within. of experimental units (e.g.
communities of users) instead of the units themselves [18]. Clustering by non-
overlapping communities means one can hold SUTVA at a cluster level, as the
units generally interact heavily within the same cluster but sparsely between
clusters [169]. However, the choice of clusters is non-trivial; this means a
cluster-based treatment assignment may only be justified when the network
effect is apparent. [239] developed a framework that detects whether network
effects exist for a particular OCE by simultaneously running the same
experiment under individual-based and cluster-based assignment and testing if
the treatment effect differs (and hence if SUTVA holds up reasonably to
justify individual-based assignment). Section 6 of [159] describes many more
relevant works, including those focusing on indirect network effects.

The second part of SUTVA (no hidden variations of treatments) is seldom
explicitly discussed in OCEs. That said, it is conceivable that many
“imperfections […] hidden throughout the engineering stack or in the design
process” [283] are capable of breaking this part of SUTVA and thus bias the
(potential) responses. Many examples are presented as pitfalls in OCE
operations, which we will cover in Section 5.4.2 / Table 5.1.

##### Improving sensitivity

We look beyond challenges that lead to biases in our treatment effect
estimator. Recall from Section 3.6.4 that the power of a test depends on many
factors, including the significance level, sampling uncertainty, sample size
and effect size. While we have discussed the role of sample and effect sizes
in Sections 3.6.5 and 3.6.6, respectively, little regarding the sampling
uncertainty has been said so far in the thesis. Some works look into
increasing the sensitivity of OCEs by reducing sampling uncertainty, which
enables an experimenter to reduce the sample size required for an experiment
or the minimum detectable effect (or both). The works are motivated by
observations where [150] noted that ordinarily, successful experiments in
technology companies only improve metrics by a fraction of a per cent; [286]
also described the need to detect small effects as huge customer bases often
translate them into substantial gains in revenue and profit.

The methods described below rely on decomposing the variance in a metric and
attempting to eliminate unnecessary components. Controlled-experiment Using
Pre-Existing Data (CUPED) is a control variates method that measures a metric
modified by a linear function of the covariates [69].202020Also known as
_regression adjustment_ in econometrics [168]. The idea is to eliminate the
effect of correlated covariates on the metric under measurement. [63] later
apply the idea to reformulate dilution212121The act of translating the
treatment effect from users who have been exposed to a treatment (“triggered”
users) to the general user base. The metric is diluted as we incorporate
untriggered users (who contribute zero). as a variance reduction problem,
resulting in a more easy to implement and less error-prone formula. A similar
idea is used by [222], which subtracted the predicted value of a metric using
boosted decision trees as predictors. In the same spirit, stratified sampling
is used to eliminate variance between strata. Stratified sampling can be
applied pre- or post-experiment, and while the theoretical bounds are better
for pre-experiment stratification, post-stratification is often preferable for
large-scale OCEs [286].

####  5.4.2 Other topics

To complete our review of OCEs, we also review topics that we did not cover in
the statistical tests and data chapters, as well as the advanced methods
section above.

##### Anatomy of an OCE

A mature OCE system/platform often features complex processes and
infrastructure, and many seek to provide an overview or describe specific
parts of the ecosystem. [154] provided a definitive guide on setting up,
running, reporting, and scaling OCEs. Several papers described the scalable,
concurrent testing infrastructure and its associated challenges at major
technology companies [149, 260, 288]. Others described the OCE lifecycle [91]
and characterised the elements defining an experiment [11].

##### State and progress of OCEs’ usage

A class of works aim to capture the state and track the progress of the use of
OCEs (or DEM capabilities in general) across many organisations. Some propose
models that represent the maturity/growth of DEM capabilities [92, 280].
Others focus on experiment outcomes and perform meta-analyses on results from
multiple OCEs – see [29, 30, 136] for results across industries, in
e-commerce, and in display advertising, respectively. [103] bridged the two
topics above by linking experiment performance to organisational structure.

##### Pitfalls in OCE operations

Despite the increasing adoption and maturity of the use of OCE, running a
successful OCE after another is, by the admission of the biggest players in
the industry, difficult [155]. This is due to many non-obvious challenges and
pitfalls experimenters face while running an OCE end-to-end. In addition to
that related to statistical testing (see Chapter 3) and requiring advanced
statistical methods (see Section 5.4.1), many accounts [53, 74, 89, 152, 148]
describe pitfalls related to experimentation infrastructure, decision metric
selection and interpretation [68], and external effects [126]. We summarise
these challenges and pitfalls in Table 5.1. To mitigate the potential
problems, [179] proposed techniques that separate problematic user responses
affected by some of the external effects (namely dilution, carry-over effect,
and novelty effect) from valid ones. [41, 93] also developed a series of
diagnostic tests to identify the root cause(s) of a sample ratio mismatch, a
common symptom for tests that are potentially invalidated due to encountering
one or more of the pitfalls (see “practical usage” paragraph of Section
3.7.2).

Table 5.1: Pitfalls in running an OCE end-to-end as described in Crook et al.
[53], Kohavi and Longbotham [152], Kohavi et al. [148], Lu and Liu [179],
Hohnhold et al. [126], Dmitriev et al. [74], Fabijan et al. [93], Chen et al.
[41], and Esteller-Cucala et al. [89]. The value in each cell represents the
section number where the pitfall is described in each work.

###  5.5 A Spotlight on Dependent Responses in e-Commerce Experiments

To complement our review of advances in OCE, we present a case study on
dependent responses in e-commerce OCEs. We intend for the case study to be a
short but crucial piece of evidence that drives awareness of the pitfall among
experimenters in e-commerce and hence encourages the adoption of established
mitigation approaches.

Like many technology organisations, e-commerce organisations aim to provide a
consistent user experience; hence, randomisation in OCEs is generally done on
a per-user basis. On the other hand, e-commerce organisations are unique as
they feature physical inventories and thus track a set of business metrics
that are transaction- or item-based. These metrics include _average basket
value_ (ABV),222222Average (mean) amount spent in each transaction/basket by a
user. _average basket size_ (ABS),232323Average (mean) number of items
purchased in each transaction/basket by a user. and _average selling price_
(ASP).242424Average (mean) price per item sold.

Experiments that measure changes to ABV, ABS, and ASP often feature dependent
responses. In these experiments, the responses (or _analysis units_) are at a
more granular transaction or item level than the _randomisation units_ , which
are at a user level. Given that a user can make many transactions and purchase
many items during an experiment, the value of these transactions and items
will likely correlate based on the user’s preference. This creates a local
dependence structure between users and transactions/items, which violates the
usual i.i.d. assumptions in common statistical tests that OCEs employ (e.g., a
Student’s $t$-test). If left unmitigated, it risks experimenters having wrong
estimates of sampling uncertainty and making wrong conclusions from a
statistical test.

Having dependent responses in experiments due to differences in granularity
between randomisation and analysis units is not a recent revelation [155]. In
some sense, one may regard such a setup as a cluster-randomised controlled
experiment, where an experimenter randomly assigns clusters of transactions or
items belonging to the same user to the same treatment groups.252525In such a
setup, the size of a cluster can be zero and would only be determined after
the experiment ends. This is because experimenters cannot control how much a
user buys on the website. In terms of obtaining an unbiased estimate for the
sampling uncertainty, many viable approaches already exist: experimenters in
medicine, economics, and social sciences often employ a crossed random effects
model [16] or cluster robust standard errors [32]. Meanwhile, those in digital
technology prefer using bootstrap [19] or the delta method [64] to estimate
the sample variance and the standard error due to their ability to scale to
large datasets and relative lack of model assumptions.

Despite the existing literature dedicated to dependent responses in digital
experiments, there is little published evidence specifically in the context of
OCEs in e-commerce. Most published research is based on the context of OCEs in
digital advertising and content, where experimenters randomise by users and
analyse by sessions or page views [19, 64]; or social networks, where
experimenters both randomise and analyse by users whose responses may
correlate via their social connections [83, 114]. We believe the lack of
evidence contributes to insufficient awareness of the issue from experimenters
in this area.

This case study’s contribution is evidence from real-life e-commerce
experimentation operations. In particular:

  1. 1.

We show, using three e-commerce transaction datasets (with two publicly
available), a positive correlation between the value/size of transactions and
items from the same user. This leads to inflation in measurement uncertainty
(Section 5.5.1);

  2. 2.

We quantify the extent of the said inflation, which can be anywhere between 1x
to $>$100x the usual estimate and is dependent on the business metric,
experiment duration, and organisation (Section 5.5.2);

  3. 3.

We highlight the impact of the said inflation on test power and confidence
interval coverage in null hypothesis significance tests (Section 5.5.3); and

  4. 4.

We share lessons learned while incorporating some of the established
mitigation approaches into an e-commerce company’s internal experiment
analysis platform, including a critique of some popular approaches and
practical tradeoffs during implementation (Section 5.5.4).

####  5.5.1 Beware of Hidden Inflation in Measurement Uncertainty

We first motivate why we may obtain inaccurate estimates of the sampling
uncertainty when measuring changes to e-commerce metrics based on transactions
or items in OCEs. Recall from the previous section (see Equation (5.13)) that
$\sqrt{s_{X}^{2}/n}$, the vanilla sample standard error (SE), is an unbiased
estimator for the SE under i.i.d. assumptions yet becomes biased when the
responses are dependent.

The i.i.d. assumption is the prevailing assumption when experimenters
randomise by users and analyse by user [67]. However, it is unlikely to hold
when experimenters randomise by users and analyse by transactions or items.
Many users tend to make a transaction that is similarly sized and valued to
their previous transaction(s). They also tend to purchase items that are at a
similar price point. To demonstrate the claims above, we utilise three online
retail/e-commerce transaction datasets. The UCI Online Retail II [40, 78] and
Olist Brazilian e-Commerce [209] datasets are publicly available.262626The
experiment code and results on the two publicly available datasets are
available on GitHub: https://github.com/liuchbryan/oce-ecomm-abv-calculation.
We also use a proprietary dataset from ASOS.com, a global online fashion
retail company, that records transactions within a specific two-month period
in 2022 on a particular mobile platform.272727The data described are not
representative of ASOS.com’s overall business operations, and one should not
draw any such conclusion from the dataset. We summarise the three datasets in
Table 5.2.

Table 5.2: Summary of the three online retail/e-commerce transaction datasets
featured in this paper (post data cleaning).

Dataset # Users/ Customers # Transactions/ Orders # Items/ Units # Products/
SKUs Time span ASOS (Proprietary) $\sim$4 180 000 $\sim$9 680 000 $\sim$32 900
000 $\sim$330 000 62 days (2 months) UCI Online Retail II [78, 40] 5 852 36
594 10 690 447 4 621 739 days (2 years) Olist Brazilian e-Commerce [209] 94
983 98 199 112 101 32 729 729 days (2 years)

For each dataset, we plot the value and size of a user’s transaction against
that of their next transaction (if it exists) in Figure 5.2. We observe from
the figures a high empirical density along the $x=y$ diagonal and strong
directionality in the kernel density estimates. This suggests a positive
correlation between transaction values/sizes from the same user. For brevity,
we do not show the plots for item values, yet contend that a perfect
correlation arises when a customer purchases multiple units of the same
product.

As responses from the same user become positively correlated, we have
$\textrm{Cov}(X_{i},X_{j})>0$ for some $i$ and $j$ in Equation (5.13) and thus
a higher SE than the vanilla sample SE.

Figure 5.2: Kernel density estimation plots showing a user’s transaction (Top
row) value / (Bottom row) size against that of their next transaction – if it
exists – in the (Left) ASOS, (Middle) UCI Online Retail II, and (Right) Olist
Brazilian e-Commerce datasets. We normalise the transaction values/sizes by
the ABV/ABS of respective datasets.

####  5.5.2 Re-estimating the SE using bootstrap

Figure 5.3: The ratio between the one-way bootstrap standard error (SE)
estimates [19] and the vanilla sample SE estimates for (Left) ABV, (Middle)
ABS, and (Right) ASP. The error bars represent the 95% confidence interval for
the one-way bootstrap SE estimates.

We now explore how much the SE can inflate. We apply the bootstrap procedure
described in Section 2.2 of [19] to the datasets described above to re-
estimate the SE. Bootstrapping generates a resample by sampling the original
set of responses with replacement (or applying a random weight in this case).
This yields a different sample mean. By repeating the process many times and
taking many bootstrap means, we can estimate the SE by calculating the
standard deviation of the bootstrap.

We use a one-way bootstrap (a.k.a. block bootstrap) to account for the
dependency between transactions/items and users. Instead of generating the
resample by sampling each transaction/item individually, we sample clusters of
transactions/items belonging to the same user, mirroring our randomisation
process.282828For ASP, in addition to the dependency between items and users,
there may be additional dependence between items and products/stock-keeping
units (SKUs). One may account for both using a two-way bootstrap described in
Section 5.5.4 or Section 2.3 of [19].  Here, we use random weights generated
from a Poisson(1) distribution to reweight our samples before calculating a
bootstrap mean. The random weights emulate the number of times a
transaction/item was sampled with replacement (along with other
transactions/items associated with the same user). Finally, we estimate the SE
from 500 bootstrap means.

Figure 5.3 shows how the one-way bootstrap SE differs from the vanilla sample
SE in successive expanding windows, corresponding to different potential OCE
durations. We observe that the bootstrap SE is significantly higher than the
vanilla sample SE, with the exact ratio between the one-way bootstrap SE and
vanilla sample SE heavily dependent on the dataset/audience, business metric,
and duration. For example, the bootstrap SE for ASP in the UCI Online Retail
II dataset is $>$100x the vanilla SE as many of their users buy tens or
hundreds of the same product. Meanwhile, the ratios for ABV and ABS in the
Olist Brazilian e-Commerce dataset are practically one. This is due to only
3.2% of transactions coming from returning users. Given such differences,
experimenters should strive to obtain a reasonably accurate estimate.

We also observe from Figure 5.4 that the bootstrap SE may no longer drop as
more transactions are involved over time and may go up again in some cases.
This is likely due to returning users making further transactions, thus
creating more and larger clusters of transaction/item values and sizes. It
leads to the response variance increasing, sometimes more quickly than the
increase in sample size. The observation suggests that lengthening an
experiment solely to collect more responses (hence lowering the SE) may
backfire. In addition to established practices on sample size estimation
[228], experimenters should also consider how the variance of their metrics
evolves when designing OCEs with dependent data.

Figure 5.4: Trajectories of the one-way bootstrap standard error (SE)
estimates (solid colored lines) and the vanilla SE estimates (dashed black
lines) under different dataset-metric combinations. Estimates are cumulative
from day one of the datasets. All x- and y-axes are on different scales.

####  5.5.3 Impact on OCE decisions

We finally discuss how an inflated SE due to dependent data can affect
decisions made from a NHST. Firstly, it reduces the power of the test, making
any potential treatment effect harder to detect. Consider a two-tailed
Student’s $t$-test with significance level $\alpha$ and $\nu$ d.f. Its power
(adapted from Equation (3.39)) is

$\displaystyle
1-T_{\nu}\big{(}t_{\nu,1-\alpha/2}-\theta/\textrm{SE}\big{)}+T_{\nu}\big{(}-t_{\nu,1-\alpha/2}-\theta/\textrm{SE}\big{)},$
(5.14)

where $\theta$ is the effect size, $T_{\nu}(\cdot)$ is the CDF, and
$t_{\nu,q}$ is the $q^{\textrm{th}}$ quantile of a $t$-distributed r.v. If the
SE increases, then both the standardised effect size $\theta/\textrm{SE}$ and
the test power in Expression (5.14) decrease.

Secondly, having an inflated SE without knowing so will lead to tests
producing confidence intervals that are too narrow, risking more false
positives. In the test above, the $(1-\alpha)$ confidence interval (CI) is
$\big{(}\bar{x}\,\pm\,t_{\nu,1-\alpha/2}\cdot\textrm{SE}\big{)}$ – see
Equation (3.4). Fixing the CI bounds, we observe that $t_{\nu,1-\alpha/2}$
must decrease (and cease to be a $(1-\alpha/2)$ quantile) when the SE
increases. This means that the said CI will no longer have a $(1-\alpha)$ but
a lower coverage, i.e., there is now a lower chance the interval will contain
the actual effect size.

Figure 5.5: (Left) The actual test power of and (Right) the coverage of the
centred confidence interval (CI) from a two-tailed $z$-test against different
standard errors (SE), expressed as a multiple of the vanilla sample SE
$\sqrt{s^{2}/n}$. We choose the test parameters such that the test power
$=80\%$ and the CI coverage $=$ the nominal values when the SE multiples are
one.

To show the full extent of the issues above, we plot the test power and CI
coverage of a two-tailed $z$-test, essentially a Student’s $t$-test with large
degrees of freedom, against different SEs in Figure 5.5. We observe that the
power of a $z$-test with a 5% significance level (dashed line) tumbles from
80% to around 29% when we merely double the vanilla sample SE. Moreover, the
centred 95% CI calculated using the vanilla sample SE would only have roughly
67% coverage. These results reinforce the importance of having an accurate
estimate of the SE.

####  5.5.4 Lessons Learned During Deployment

We finally share some lessons learned when incorporating the calculations of
e-commerce metrics into ASOS.com’s experiment analysis platform, which has
been in production since Sep 2019. The analysis platform extends the
functionality of a third-party experimentation platform, namely splitting,
logging, and alerting. It adds value to the business by computing and
performing statistical tests on bespoke business metrics that use internal
datasets.

As mentioned at the beginning of the section, experimenters in digital
technology prefer the use of bootstrap (see Section 2.2 of [19]) and the delta
method (see Section 3 of [64]) to re-estimate the SE, which acts as a plug-in
estimate for subsequent statistical tests. We described and implemented the
bootstrap approach in Section 5.5.2 when we quantified the extent of the SE
inflation. The delta method approach expresses a transaction-/item-based
metric as the quotient of two user-based metrics. For example, we can express
ABV as the quotient of the business metrics “mean spend per user (across all
baskets)” and “mean number of baskets per user.” Given that both metrics in
the quotient are asymptotically normal and based on randomisation unit-based
responses, which are commonly assumed to be i.i.d. [67], we can use the delta
method to estimate the variance of the quotient. The resultant formula has an
explicit term for the dependency between users and transactions/items.

##### Delta method - a straightforward formula with hidden complexity and
inflexibility

The delta method approach is more compute efficient than the bootstrap
approach. It requires passing only group-level statistics around instead of
analysis/sub-randomisation unit-level responses required by bootstrap
methods.292929That said, to calculate the group-level statistics, which
include the covariance between users and transactions/items, one still
requires _access_ to sub-randomisation unit responses initially (see Section
4.4.1). However, it is unclear how the results in [64], developed with a one-
way dependency between randomisation and analysis units in mind, would apply
to business metrics that feature responses dependent on other types of
units.28 The approach also requires considerable statistical mastery to
understand and implement, risking decreased engagement from engineering teams
and non-technical stakeholders due to general apprehensiveness towards “black
box” algorithms.

##### Multi-way bootstrap may be overkill by being overly conservative

The multi-way bootstrap described in Section 2.3 of [19] aims to address
dependencies between different types of units. One may apply such an approach
to item-based metrics (e.g., ASP) calculations, where dependence between items
and products exists alongside dependence between items and users.

Recall from Section 5.5.2 that in a one-way reweighting bootstrap, we apply a
random weight drawn from Poisson(1) to each user and all their associated
items before calculating a single bootstrap mean. In a two-way bootstrap, we
apply a random weight to each user and a separate random weight to each
product. The final weight for each item is the product of the user weight and
the product weight.303030In the general case where there are dependencies
between multiple types of units, we apply a random weight to each unit within
a single unit type, repeat for other unit types with a separate set of random
weights, and take the final weight of each link/interaction/dependency between
the unit types as the product of the weight of their associated units. Similar
to the one-way bootstrap, the final weight emulates the number of times the
item will be selected if we separately sample the users and products with
replacement. The calculations for both the bootstrap means and SE remain
identical post-reweighting.

As a two-way bootstrap preserves the correlation structure between items,
users, and products during sampling, one would expect the method to generate
less biased (if not unbiased) SE estimates. However, the SE estimates from a
two-way bootstrap are impractically large (see Table 5.3). This appears to
match the results reported in [19], where two-way bootstrap SE estimates
generally yield CIs that are overly conservative in A/A tests.313131A/A tests
are tests featuring two supposedly identical treatments and, thus, produce
results that are consistent with a nil null hypothesis. See Section 3.2 for
further discussions.  While the true SE will always be unknown, we believe
overestimating it is just as bad as underestimating it, as experimenters will
then struggle to design any experiments with sufficient power.

The analysis platform currently uses a one-way, user-based bootstrap to
estimate the SE of e-commerce metrics. The approach mimics the randomisation
process, and there is extensive evidence (both in [19] and within the
organisation) that it produces CIs with the right coverage in A/A tests. The
team responsible constantly evaluates whether a multi-way bootstrap will
become necessary, especially for experiments with product-treatment
interactions.323232Section 4 of [19] shows that a one-way, user-based
bootstrap may underestimate the SE of item-based metrics when there are
product-treatment interactions. Note that “products” in this case study are
referred to as “items” in [19] due to different business contexts.

Table 5.3: The ratio between the bootstrap standard error (SE) estimates and the vanilla sample SE estimates for ASP. Each entry represents the mean and standard deviation of the ratios across multiple experiment runs. Dataset | Duration | One-way bootstrap | Two-way bootstrap
---|---|---|---
ASOS | First 30d | 3.51 $\pm$ 0.13 | 17.77 $\pm$ 0.56
UCI | First 10% | 64.45 $\pm$ 1.94 | 72.83 $\pm$ 2.20
First 50% | 87.95 $\pm$ 2.82 | 105.15 $\pm$ 3.33
All | 69.82 $\pm$ 1.37 | 94.33 $\pm$ 2.28
Olist | First 20% | 1.104 $\pm$ 0.035 | 2.083 $\pm$ 0.075
First 50% | 1.058 $\pm$ 0.036 | 2.531 $\pm$ 0.066
All | 1.076 $\pm$ 0.037 | 3.364 $\pm$ 0.109

##### Further consideration - error vs runtime/cost tradeoff

Bootstrap methods introduce their error (more precisely, _variance component_)
to SE estimates during the resampling process. This is in addition to the
(inflated) sampling uncertainty and generally scales along
$O\big{(}1/\sqrt{B}\big{)}$, where $B$ is the number of bootstrap resamples
[86]. Given the runtime of bootstrap methods scales along $O(B)$, at some
point, it is no longer economically practical for a production system to draw
further resamples to reduce the error to SE estimates.

Deciding where the tipping point is more an art than science and depends more
on individual needs. Unlike many technology giants which perform analysis in a
streaming fashion, our analysis platform features batch-processing pipelines
due to temporal constraints from upstream data sources. As a result, our
priority is to keep the processing jobs’ runtime (and hence compute cost) low.
The team finds $B\in[500,1000]$ works well as it leads to a coefficient of
variation (error of SE estimate divided by mean SE estimate) of $<5\%$. That
said, we believe the tradeoff also applies to streaming systems as the compute
latency increases with a larger $B$, which impacts system availability.

###  5.6 Advances in Quasi-experiments and Natural Experiments

We move on to works related to quasi-experiments and natural experiments.
While they differ regarding whether an experimenter can impose scientific
control, they involve the same set of causal inference techniques to estimate
the treatment effect. We thus combine the discussion on related methods and
examples for the two experiment classes.

To date, few works are dedicated to reviewing the use of causal inference
techniques in digital experimentation. Chapter 11 of [155] briefly outlined
commonly used causal inference methods with examples, and [62] is a work-in-
progress discussing its application in the online industry. That said, causal
inference is an established discipline, and many excellent introductions and
reviews exist:

  * •

[122] and [131] provide an accessible introduction to the area and review some
models and techniques. Both books stem from courses the respective authors
taught in the past decades and are geared towards researchers and
practitioners in the medical and social sciences, the latter sharing some
scientific goals with digital experimentation.

  * •

[216] focuses on the theoretical constructs (or representations) in causal
inference – structural equation models, the potential outcomes framework (see
Section 5.2 for the basics), and graphical models. The paper also proposed the
structural causal model, which is seen as either a unifying or an alternative
representation to those above.

  * •

[290] extensively surveys the methods, datasets, software packages, and
applications in (general) causal inference. Sections of note include the
applications in advertising and recommendations (Sections 6.1–6.2). The paper
is tailored to the computer science and data mining communities, which
substantially overlap with the digital experimentation community.

  * •

[140] explores works in the intersection of causal inference and machine
learning (CausalML), which has flourished in the past decade. At the same
time, [267] outlines many of its applications in economics and marketing. Many
experiments described in the latter are run offline, though most techniques
can be directly translated to an online setting.

We keep our review close to digital experimentation and cover the following in
the remainder of the section. We first revisit the conditional average
treatment effect (CATE) and conditional exchangeability concepts in Section
5.6.1. Many causal inference methods seek to estimate the former and assume
the latter. We then list several classes of popular methods in causal
inference, followed by examples applying the said methods in digital
experiments in Section 5.6.2. We finally provide a mini-spotlight in geo-
experiments, a form of quasi-experiments that assign treatments based on the
geographical region a user falls within, in Section 5.6.3.

####  5.6.1 CATE and Conditional Exchangeability Revisited

Many causal inference techniques seek to estimate the conditional average
treatment effect (CATE), the average treatment effect among those sharing
certain covariates $\mathcal{C}$ (those on which the treatment assignment
$\mathcal{A}$ does not affect):333333Here, we separate the covariates
$\mathcal{C}$ and the treatment assignment indicator $\mathcal{A}$. Some in
the causal inference community lump these two objects together when discussing
CATE. Others may have a strict separation between these objects, with
additional terminologies such as average treatment effect on the treated (ATT,
i.e., CATE on ${\mathcal{A}=1}$) and average treatment effect on the
control/untreated (ATC/ATU, i.e. CATE on $\mathcal{A}=0$) [110].

$\displaystyle\mathbb{E}(Y(1)-Y(0)\,|\,\mathcal{C})=\mathbb{E}(Y(1)\,|\,\mathcal{C})-\mathbb{E}(Y(0)\,|\,\mathcal{C})\,.$
(5.15)

The availability of covariates means one can produce a more expressive model
$Y=g(\mathcal{A},\,\mathcal{C})$ to capture the complex dynamics between the
treatments and observed responses, achieved by drawing on the wealth of
potentially complex statistical and machine learning models [137]. Of course,
such a model may still only capture correlation/association instead of
causation, as its dependent/target variables are not the potential responses.

If, in addition, the covariates identify and account for all confounders, we
noted that the potential responses are conditionally exchangeable, i.e.,
$Y(0),Y(1)\perp\kern-6.00006pt\perp\mathcal{A}\,|\,\mathcal{C}$.343434Also
known as _conditional unconfoundedness_ or _strong ignorability_ [62, 235,
240]. Similar to its unconditional counterpart in Equation (5.6), if the
potential responses are conditionally exchangeable, we have

$\displaystyle\mathbb{E}(Y(a)\,|\,\mathcal{C})=\mathbb{E}(Y(a)\,|\,\mathcal{A}=1,\,\mathcal{C})=\mathbb{E}(Y(a)\,|\,\mathcal{A}=0,\,\mathcal{C}),\;a\in\\{0,1\\}\,.$
(5.16)

This, combined with the definition of potential outcomes in Equation (5.3),
enables us to estimate the potential responses using the observed responses
$Y$ conditioned on the covariates $\mathcal{C}$ and, thus, develop a model
that captures the complex dynamics between the treatments and potential
responses, i.e., causation.

At first glance, conditional exchangeability appears to be a huge assumption.
This is mostly true but not insurmountable in practice. We live in a complex
and dynamic environment; thus, it is difficult to identify and account for all
confounders, which biases the treatment effect estimate as discussed in
Section 5.2. On the other hand, many environmental variables have a tenuous
link to our treatments and responses, and the bias they introduce may be
practically insignificant. Identifying confounders is one of the main tasks in
causal discovery and remains an active research topic, together with the study
of the impact of hidden confounders [108, 217].

It is also worth emphasising that the decision to estimate CATE instead of ATE
and the ability to perform random assignments are two orthogonal concepts. One
can estimate CATE in randomised controlled trials (as in the case of
heterogeneous treatment effect detection and uplift modelling discussed in
Sections 5.4.1 and 4.3). They can also recover the population-level ATE from
various CATE estimates by applying the law of total expectation:

$\displaystyle\mathbb{E}(Y(1)-Y(0))=\mathbb{E}_{\mathcal{C}}(\mathbb{E}(Y(1)-Y(0)\,|\,\mathcal{C})).$
(5.17)

That said, it might be difficult in practice as the distribution of
$\mathcal{C}$ is often unknown. While one can use the empirical distribution
of $\mathcal{C}$, it may incur bias.

####  5.6.2 Methods and Examples

Given the underlying goal, we are in a position to list and briefly describe
some popular causal inference techniques. [6] called the following approaches
the “furious five” in econometrics, most of which are translatable to digital
experiments:

  * •

Randomisation: This has been extensively discussed in the previous sections.

  * •

(Outcome) regression: A model that maps the relationship between the treatment
assignment indicator, other covariates, and the responses. By manipulating the
treatment assignment indicator within the fitted model, one can obtain the
CATE for a particular covariate combination, provided the covariates present
are sufficient for conditional exchangeability. See Chapter 9 of [100] for
further discussions.

  * •

Instrumental variables: Variables that only affect the responses via their
effect on the treatment assignment indicator or other covariates (but not
directly). They must also be independent of the error term. These variables
are often employed when the treatment assignment indicator or other covariates
correlate with the unobserved error terms in a regression model.

  * •

Regression discontinuity design: Specifies a cut-off to a continuous
eligibility index and assigns treatment based on which side of the cut-off an
experimental unit falls on. By exploiting the assumption that there is no
meaningful difference in experimental units’ characteristics or activity
around the cut-off point before the treatment, one can estimate the local
treatment effect as all others are roughly equal.

  * •

Difference-in-differences: Calculates, per treatment, the difference in
average response pre- and post-treatment and, then, another difference in the
differences arising from different treatments. Under the parallel trend
assumptions, i.e., the non-treatment variables affect the average response in
both groups similarly across time, one is constructing a counterfactual when
taking the per-treatment difference and estimating the treatment effect by
comparing the counterfactual to the observed when taking the final difference.

##### Propensity scores

To address hidden confounders between treatment and outcomes, some
experimenters match the experimental units or reweigh the responses. This is
usually done via the _propensity score_ , the probability that an experimental
unit is assigned a certain treatment given the covariates associated with the
unit [235]. The score is most straightforwardly estimated via a logistic
regression model. Once calculated, it is used to match a unit receiving a
treatment with other units receiving another treatment (known as _propensity
score matching_) [235]. The estimated treatment effect is the difference in
responses between the unit and its matched counterparts, averaged over all
matches. Alternatively, one can reweigh the responses using the score and take
the difference between the reweighted average response to obtain the treatment
effect (known as _inverse probability/propensity weighting_) [121].

Another increasingly popular technique related to propensity scores is _doubly
robust estimation_ [98]. It combines outcome regression and propensity score
models and yields a valuable property: the treatment effect estimates are
consistent when at least one of the two models is correctly specified. This
approach is taken by [287] to estimate the effect of changes to LinkedIn’s
mobile app, where the ability to perform random assignments is surrendered to
the app store and users (as discussed in Section 5.3.2).

##### Mediation analyses

Digital experimenters also turn to mediation analyses. They feature the use of
mediators, variables directly influenced by the treatment assignment, and, in
turn, directly influence the responses.353535Not to be confused with
instruments. Explaining _why_ and _how_ a mediator and an instrument are
different involves an introduction to graphical models in causal inference,
which we have omitted for brevity. See, e.g., Chapters 2 & 3 of [218] for more
details. Using such intermediary variables enables one to attribute it to
different causal pathways between the treatment assignment and change in
responses. It also enables one to produce more accurate treatment effect
estimates if the variables are unaffected by the same confounders that affect
the treatment assignment indicator and responses. This is the case in [123],
who, seeking to measure the effect of a display advert targeting algorithm on
certain user behaviour, uses ad viewability363636Defined in [123] as whether
at least half of the advert has come to a user’s view for longer than a
second. Treatment in display advertising is usually defined as serving an
advert, i.e., the advert is planned/scheduled to appear on a user’s screen as
they scroll through the page. While such a definition leads to much easier
tracking from an experimenter’s perspective, the ad may not come into users’
view for many reasons (e.g., the user stops scrolling or has the wrong browser
dimensions). [123] observed that 45% of adverts have not come to view and
hence cannot influence any change in user behaviour. as a mediating event.
Moreover, [274], in the quest to measure the impact of algorithmic changes to,
e.g., predictive models or recommender systems, uses evaluation metrics from
online A/B tests as mediators. They bridge the algorithmic changes’ effect on
evaluation metrics in offline experiments to that on business KPIs.

####  5.6.3 A mini spotlight on geo-experiments

A class of quasi-experiments gaining traction in the past decade is geo-
experiments. They are heavily used in display advertising [20, 268] – the
author also led a team at ASOS.com to build and maintain such a geo-
experimentation platform to help its marketing function manage tens of
millions of spend [17, 188].

A geo-experiment assigns treatments at a geographical region level. For
example, it may assign all users based in Oxfordshire to control and all users
based in Cambridgeshire to treatment. Such treatment assignment is
necessitated by the inability to target and track individual users (due to
privacy and data protection concerns or third-party platform restrictions)
and, sometimes, by interference between individual users (see Section 5.4.1).
The treatment assignment may or may not be random – while a randomised
assignment will simplify the experiment to a straightforward cluster
randomised OCE (see Section 5.4.1), experimenters may decline to do so due to
various operational constraints (e.g., cross-region user activities, minimum
advertising budget/coverage requirements).

We estimate the treatment effect by comparing the responses from regions
receiving the treatment to a counterfactual, constructed using the responses
from control regions, on how the same regions would have performed had there
been no treatment. More precisely and in the language of potential outcomes,
consider a simplified case where we have two regions, one with response $X$
and the other with response $Y$. We call the former the control region and the
latter the treatment region. We also assume we can access some covariates
$\mathcal{C}$ shared between the two regions that address all confounding,
i.e., conditional exchangeability
$Y(0),Y(1)\perp\kern-6.00006pt\perp\mathcal{A}\,|\,\mathcal{C}$ holds.

Before the experiment-proper, we assign both regions to $\mathcal{A}=0$ (i.e.,
control) and observe

$\displaystyle X\,|\,\mathcal{A}=0,\,\mathcal{C}\quad\textrm{ and }\quad
Y\,|\,\mathcal{A}=0,\,\mathcal{C}\,.$ (5.18)

We know from the definition of potential outcomes (see Section 5.2 / Equation
(5.3)) that Expression (5.18) is equivalent to

$\displaystyle X(0)\,|\,\mathcal{A}=0,\,\mathcal{C}\quad\textrm{ and }\quad
Y(0)\,|\,\mathcal{A}=0,\,\mathcal{C}\,.$ (5.19)

Suppose we have a mapping $g(\cdot)$ between the two observations under the
same treatment assignment using the covariates, i.e.,
$Y=g(X,\mathcal{A}=0,\,\mathcal{C})$. Given that the observations in
Expressions (5.18) and (5.19) are equivalent, the following must also be true:

$\displaystyle Y(0)=g(X,\mathcal{A}=0,\,\mathcal{C})\,.$ (5.20)

During the experiment, we assign the control region to $\mathcal{A}=0$ and the
treatment region to $\mathcal{A}=1$ and observe373737Readers may notice that
it seems possible to observe responses under _both_ treatments
($Y\,|\,\mathcal{A}=0,\,\mathcal{C}$ in Expression (5.18) and
$Y\,|\,\mathcal{A}=1,\,\mathcal{C}$ in Expression (5.21)), which contradicts
the premise of counterfactual reasoning. There is merely an artefact of
oversimplifying the notations for illustration purposes – the two observed
responses happened at different points in time and, thus, ought to be
represented by different r.v.’s. Here, we pretend
$Y\,|\,\mathcal{A}=0,\,\mathcal{C}$ was never observed when we constructed the
counterfactual.

$\displaystyle X\,|\,\mathcal{A}=0,\,\mathcal{C}\quad\textrm{ and }\quad
Y\,|\,\mathcal{A}=1,\,\mathcal{C}\,.$ (5.21)

To estimate $Y(0)$, we require constructing a counterfactual using the
observation $X$ and the mapping $g(\cdot)$, i.e.
$g(X,\mathcal{A}=0,\,\mathcal{C})\,|\,\mathcal{A}=0,\,\mathcal{C}$. This
enables us to take the expected difference between the observed $Y$ and the
counterfactual to obtain the CATE:

$\displaystyle\mathbb{E}(Y\,|\,\mathcal{A}=1,\,\mathcal{C})-\mathbb{E}(g(X,\mathcal{A}=0,\,\mathcal{C})\,|\,\mathcal{A}=0,\,\mathcal{C})$
$\displaystyle=\,$
$\displaystyle\mathbb{E}(Y\,|\,\mathcal{A}=1,\,\mathcal{C})-\mathbb{E}(Y(0)\,|\,\mathcal{A}=0,\,\mathcal{C})$
(RHS: By Eq. (5.20)) $\displaystyle=\,$
$\displaystyle\mathbb{E}(Y(1)\,|\,\mathcal{A}=1,\,\mathcal{C})-\mathbb{E}(Y(0)\,|\,\mathcal{A}=0,\,\mathcal{C})$
(LHS: By definition – Eq. (5.3)) $\displaystyle=\,$
$\displaystyle\mathbb{E}(Y(1)\,|\,\mathcal{C})-\mathbb{E}(Y(0)\,|\,\mathcal{C})$
(Conditional exchangeability – Eq. (5.16)) $\displaystyle=\,$
$\displaystyle\mathbb{E}(Y(1)-Y(0)\,|\,\mathcal{C})$
$\displaystyle\textrm{(Linerarity of conditional expectation)}\,.$ (5.22)

The example above is grossly simplified. In reality, there are many copies of
$X$ and $Y$, both along the geographical (responses from multiple regions) and
time dimensions (responses at multiple time points). Given the many variables
involved, experimenters often prefer using the more expressive structural
equation model (instead of the potential outcomes framework) to express causal
assumptions [20, 268].

Many models have been proposed for the mapping $g(\cdot)$, with the two most
common being geo-based regression (GBR) and time-based regression (TBR)
models. GBR uses the aggregated metric value of the regions before and during
the experiment, plus the advertising spend as covariates [268, 269]. Thus, it
allows experimenters to easily extract the return on ad spend (ROAS) metric
from the fitted model. However, the statistical power of GBR is derived from
the number of regions involved, which is problematic when the number of
regions available for an experiment is small. The challenge is mitigated by
TBR, which uses the value of the covariates at each time interval. TBR also
employs Bayesian regression techniques, more precisely, Bayesian structural
time series. This allows access to the full posterior distribution of the
counterfactual and, thus, the calculation of a credible interval [39, 145].

That said, similar to many other Bayesian models, one needs to take care when
specifying the prior distributions. The team at ASOS.com found during
development that non-informative priors generate credible intervals
(associated with the posterior distribution of the counterfactual response or,
more generally, the posterior predictive) that are too narrow based on A/A
tests.31 The team thus switched to informative priors that enabled them to
calibrate the variance of the posterior predictive based on simulated
experiments.

There are many other considerations in designing a geo-experiment. They
include (1) the creation of regions and (2) the selection of regions into
control and treatment groups. Often, regions are handcrafted based on expert
input, though [232] showed it can be learnt from clustering past user queries
using clickstream data. The optimal allocation of the regions into control and
treatment groups is studied in [20] and [9], with the former focusing on
finding the best region pair (i.e. one region each for control and treatment)
and the latter finding the best disjoint subsets of regions
available383838Consider the case where the experimenter can assign treatments
over regions $\\{1,2,3,\cdots,7\\}$. The optimal allocation may assign
$\\{2,4,7\\}$ to control and $\\{3,5\\}$ to treatment, leaving $\\{1,6\\}$
unused. using a hill-climbing algorithm.

Finally, an experimenter also needs to consider the duration for different
stages of a geo-experiment [17, 188]. Unlike most digital OCEs, geo-
experiments feature multiple stages: two – the period before the experiment
where the data arising from two identical treatments is used to train the
model and the period where the treatments differ – or more if an experimenter
imposes one or more “cooldown” periods to avoid conflating effect from
multiple treatments across time. The test power of geo-experiments is also
less dependent on the sample size but more on the model fit, which reflects
the variance of the counterfactual responses. This complicates power, sample
size (or duration), and minimum detectable effect calculations. Often, one
performs a grid search across all possible combinations of stage durations to
determine the optimal experiment runtime.

###  5.7 A Brief Recap

We presented a brief review of recent advances in digital experimentation.
These include advanced statistical methods that address the many sources of
bias (e.g., post-selection inference, heterogeneous treatment effects, SUTVA
violations) to experiments, improve experiment sensitivity, and produce
treatment effect estimates using general causal inference techniques when one
cannot perform a randomised controlled trial. To organise the research
reported in the chapter, we have also introduced the potential outcomes
framework and classified experiments into online randomised controlled trials,
quasi-experiments, and natural experiments depending on the level of control
an experimenter has on performing random assignments and imposing scientific
control.

Interleaved between the reviews is an original case study on dependent
responses ine-commerce OCEs, which arises as experimenters randomise by users
to ensure a consistent user experience yet analyse by a more granular
transaction- or item-based decision metric as is common in the industry. Using
three real online retail/e-commerce transaction datasets, we provided evidence
of the extent of the dependent responses in practice. We then highlighted its
impact on test power and confidence interval coverage. Finally, we shared the
lessons learned when incorporating relevant methods in the review above into
the internal experiment analysis platform of ASOS.com, a global online fashion
retail company. We also provided a spotlight on geo-experiments, an
increasingly popular experimental design to measure the effect of display
advertising.

##  Chapter 6 An Evaluation Framework for Personalisation Strategy Experiment
Designs

This chapter is adapted from the research paper “An Evaluation Framework for
Personalization Strategy Experiment Designs”, presented at and awarded Best
Student Paper of AdKDD 2020 Workshop (in conjunction with SIGKDD ’20) [176].

###  6.1 Motivation

Without mentioning experimental design, no discussion on statistical
challenges in digital experimentation and measurement capabilities is
complete. As alluded to in Section 5.3.4, in this chapter, we focus on the
design of experiments that compare _personalisation strategies_ – complex sets
of targeted user interactions executed by e-commerce and digital marketing
organisations that aim to create an individualised experience for every user
to their websites. In addition to that mentioned in Section 5.3.4, examples of
personalisation strategies include the scheduling, budgeting, and ordering of
marketing activities directed at a user based on their browsing and purchase
history.

Compared to one’s usual online controlled experiment (see Section 5.3.1 for
examples), experiments for personalisation strategies face two unique
challenges. Firstly, strategies are often only applicable to a small fraction
of the user base, leading to many simple experiment designs suffering from
either a lack of test power due to low sample size or a diluted treatment
effect due to the inclusion of irrelevant samples [63].

Secondly, users are not randomly assigned _a priori_ but must qualify to be
treated with a strategy via their actions or attributes. This leads to a one-
sided non-compliance problem where experimenters do not have complete control
over applying scientific control [135]. At best, they can randomise a user’s
eligibility to be treated and treat those who are both eligible and qualified
[135, 270]. Under such a situation, groups of users subjected to different
strategies cannot be assumed statistically equivalent and, hence, are not
directly comparable.

While several variance reduction techniques, including stratification and
control variates [69, 222] (see Section 5.4), partially address the
challenges, the strata and control variates can vary dramatically from one
personalisation strategy experiment to another, requiring many _ad hoc_
adjustments. As a result, such techniques may not scale well when
organisations design and run hundreds or thousands of experiments at any given
time.

We argue that personalisation strategy experiments should focus on the
assignment of users from the strategies they qualified for to the
treatment/analysis groups. We call this mapping process an _experiment setup_.
Identifying the best experiment setup increases the chance of detecting any
treatment effect. An experimentation framework can also quickly reuse and
switch between different setups with little custom input, ensuring the
operation can scale. More importantly, the process does not hinder the
subsequent application of variance reduction techniques, meaning we can still
apply the techniques post hoc if required.

To date, many experiment setups exist to compare personalisation strategies.
An increasingly popular approach is to compare the strategies using multiple
control groups – Quantcast calls it a dual control [24], and Facebook calls it
a multi-cell lift study [172, 270]. In the two-strategy case, this involves
running two experiments on two random partitions of the user base in parallel,
with each experiment further splitting the respective partition into
treatment/control and measuring the _incrementality_ (the change in a business
metric as compared to the case where we do nothing) of each strategy. We then
compare the incrementalities of the two strategies against each other.

Despite the setup above gaining traction in display advertising, there is a
lack of literature on whether it is a better setup – one with a higher
sensitivity, presents a higher effect size, or both. While [172] noted that
multi-cell lift studies require many users, they did not discuss how the
number compares to other setups.111A single-cell lift study is often used to
measure the incrementality of a single personalisation strategy and, hence, is
not a representative comparison. Identifying and adopting a better experiment
setup can reduce the required sample size and enable more cost-effective
experimentation.

We address the gap in the literature by introducing an evaluation framework
that compares experiment setups given two personalisation strategies. The
framework is designed to be flexible in dealing with a wide range of baselines
and changes in user responses presented by any pair of strategies
(_situations_ hereafter). However, we also recognise the need to quickly
compare typical setups and provide some simple rules of thumb for situations
where one setup will be better. In particular, we outline the desirable
conditions for employing treatment effect dilution and a multi-cell setup.

To summarise, our contributions are:

  1. 1.

(Section 6.2) We develop a flexible evaluation framework for personalisation
strategy experiments, where one can compare two experiment setups given the
situation presented by two competing strategies;

  2. 2.

(Section 6.3) We provide simple rules of thumb to enable experimenters who do
not require the full flexibility of the framework to compare typical setups
quickly; and

  3. 3.

(Section 6.4) We make our results useful to practitioners by making the code
that performs empirical verification publicly available.222The code is
available on GitHub:
https://github.com/liuchbryan/experiment_design_evaluation.

###  6.2 Evaluation Framework

We first present our evaluation framework for personalisation strategy
experiments. The experiments compare two personalisation strategies, which we
refer to as Strategy 1 and Strategy 2. Often, one is the existing strategy,
and the other is a new strategy we intend to test and learn from. In this
section, we introduce

  1. 1.

How users qualifying themselves into strategies creates non-statistically
equivalent groups,

  2. 2.

How experimenters usually assign the users, and

  3. 3.

When we consider an assignment to be better.

####  6.2.1 User grouping

As users qualify themselves into the two strategies, four disjoint groups
emerge: those who qualify for neither strategy, those who qualify only for
Strategy 1, those who qualify only for Strategy 2, and those who qualify for
both strategies. We denote these groups (user) Groups 0, 1, 2, and 3,
respectively (see Figure 6.1). It is perhaps obvious that we cannot assume
those in different user groups are statistically equivalent and compare them
directly.

Figure 6.1: Venn diagram of the user groups in our evaluation framework. The
outer, left inner (red), and right inner (blue) boxes represent the entire
user base, those who qualify for Strategy 1, and those who qualify for
Strategy 2, respectively.

We assume Groups 0, 1, 2, and 3 have $n_{0}$, $n_{1}$, $n_{2}$, and $n_{3}$
users, respectively. We also assume responses from users (which we aggregate,
often by taking the mean, to obtain our business metric) are distributed
differently between groups, and within the same group, between the scenario
where the group is subjected to the treatment associated with the
corresponding strategy and where nothing is done (baseline). We list all
group-scenario combinations in Table 6.1 and denote the mean and variance of
the responses $(\mu_{G},\sigma^{2}_{G})$ for a combination $G$.333For example,
responses for Group 1 without any interventions have mean and variance
($\mu_{C1}$, $\sigma^{2}_{C1}$), and that for Group 2 with the treatment
prescribed under Strategy 2 have mean and variance ($\mu_{I2}$,
$\sigma^{2}_{I2}$).

Table 6.1: All group-scenario combinations in our evaluation framework for personalisation strategy experiments. The columns represent the groups described in Figure 6.1. The baseline represents the scenario where we do nothing. We assume those who qualify for both strategies (Group 3) can only receive treatment(s) associated with either strategy. | Group 0 | Group 1 | Group 2 | Group 3
---|---|---|---|---
Baseline (Control) | $C0$ | $C1$ | $C2$ | $C3$
Under treatment (Intervention) | $/$ | $I1$ | $I2$ | $\begin{array}[]{ll}\textrm{Under Strategy 1:}\;I\phi\\\ \textrm{Under Strategy 2:}\;I\psi\end{array}$
Figure 6.2: Experiment setups overlaid on the user grouping Venn diagram in
Figure 6.1. The hatched boxes cover users included in the analysis, and the
downward triangles and dots cover users subjected to the treatment(s)
prescribed under Strategies 1 and 2, respectively. See Section 6.2.2 for a
detailed description.

####  6.2.2 Experiment setups

Many experiment setups exist and are in use in different organisations. Here,
we introduce four typical setups of various sophistication, which we also
illustrate in Figure 6.2.

##### Setup 1 (Users in the intersection only)

The setup considers only users who qualify for both strategies. The said users
are randomly split (usually 50/50) into two (analysis) groups, $A$ and $B$,
and are prescribed the treatment specified by Strategies 1 and 2,
respectively. The setup is easy to implement, though it is difficult to
translate any learnings from the experiment to other user groups (e.g., those
who qualify for one strategy only) [74].

##### Setup 2 (All samples)

The setup is a simple A/B test that considers all users, regardless of whether
they qualify for any strategy. The users are randomly split into two analysis
groups, $A$ and $B$, and are prescribed the treatment specified by Strategy
1(2) if (i) they qualify under the strategy and (ii) they are in Group
$A$($B$). This setup is easiest to implement but usually suffers severely from
dilution in treatment effect [63].

##### Setup 3 (Qualified users only)

The setup is similar to Setup 2, except only those who qualified for at least
one strategy (“triggered” users in some literature [63]) are included in the
analysis groups. The setup sits between Setup 1 and Setup 2 in terms of user
coverage. It has the advantage of capturing the greatest number of relevant
samples yet having the least treatment effect dilution. However, the setup
also prevents one from telling the incrementality of a strategy itself, but
only the difference in incrementalities between the two strategies.

##### Setup 4 (Dual control / multi-cell lift test)

As described in Section 6.1, the setup first split the users randomly into two
randomisation groups. For the first randomisation group, we consider those who
qualify for Strategy 1 and split them into Analysis Groups $A1$ and $A2$.
Group $A2$ receives the treatment prescribed under Strategy 1, and Group $A1$
acts as a control. The incrementality for Strategy 1 is then the difference in
the business metric between Groups $A2$ and $A1$. Next, We apply the same
process to the second randomisation group, with Strategy 2 and Analysis Groups
$B1$ and $B2$ in place. Finally, we compare the incrementality for Strategies
1 and 2. The setup allows one to obtain the incrementality of each strategy
and minimises treatment effect dilution. However, it also leaves many samples
unused and creates additional analysis groups, and hence generally suffers
from a low test power [172].

####  6.2.3 Evaluation criteria

There are several considerations when one evaluates competing experiment
setups. These include technical considerations, such as the complexity of
setting up the setups on an OCE framework, and business considerations, such
as whether the incrementality of individual strategies is required.

Here, we focus on the statistical aspect and propose two evaluation metrics:

  1. 1.

The actual average treatment effect size as presented by the two analysis
groups in an experiment setup, and

  2. 2.

The sensitivity of the experiment setup, represented by the minimum detectable
effect (MDE) under a pre-specified test power.

Both evaluation metrics are necessary: the former indicates whether a setup
suffers from treatment effect dilution, whereas the latter indicates whether
the setup suffers from a lack of power or sample size.444The evaluation metric
on experiment sensitivity pertains to the _efficiency_ of a statistical test
[254]. However, existing literature on this topic often compares the minimum
sample size required by competing statistical tests instead. As shown earlier
in Section 3.6, under a given significance level $\alpha$ and power threshold
$\pi_{\textrm{min}}$, fixing the effect size will determine the sample size
required and vice versa. Therefore, we opt to compare the MDE of competing
personalisation strategy experiment designs because (1) it is generally easier
to estimate the sample size and (2) it is on the same scale as the other
evaluation metric. An ideal setup should yield a high actual effect size and a
high sensitivity (i.e., a low MDE),555We will use the terms “high(er)
sensitivity” and “low(er) MDE” interchangeably. though as we observe in the
next section, it is usually a trade-off.

We formally define the two evaluation metrics. Let $A$ and $B$ be the two
analysis groups in an experiment setup, with user responses randomly
distributed with independent mean and variance $(\mu_{A},\sigma^{2}_{A})$ and
$(\mu_{B},\sigma^{2}_{B})$, respectively. We also assume the sample size,
namely $n_{A}$ and $n_{B}$ for Groups $A$ and $B$, respectively, is sufficient
such that the mean response of the two groups approximately follows the normal
distribution by the central limit theorem. This enables us to run a two-sample
approximate $z$-test.

The actual effect size is simply the difference in the mean of the two
analysis groups:

$\displaystyle\Delta\triangleq\mu_{B}-\mu_{A}\,.$ (6.1)

It is worth noting that, unlike the means and variances of the user groups,
the response means of the analysis groups, thus $\Delta$, vary depending on
the experiment setup. We will demonstrate this property in Section 6.3.1.

As covered in Section 3.6.6, the MDE of a two-sample, two-sided $z$-test is

$\displaystyle\theta^{*}\triangleq\left(z_{1-\frac{\alpha}{2}}-z_{1-\pi_{\textrm{min}}}\right)\,\sqrt{\frac{\sigma^{2}_{A}}{n_{A}}+\frac{\sigma^{2}_{B}}{n_{B}}}\,.$
(6.2)

We finally define what it means to be better under these evaluation metrics.
WLOG, we assume the actual effect sizes of the two competing experiment setups