# YEAST: Yet Another Sequential Test

Alexey Kurennoy<EMAIL_ADDRESS>ZalandoDublinIreland ,  Majed
Dodin<EMAIL_ADDRESS>ZalandoBerlinGermany ,  Tural Gurbanov
<EMAIL_ADDRESS>ZalandoBerlinGermany and  Ana Peleteiro Ramallo
<EMAIL_ADDRESS>ZalandoBerlinGermany

(2024)

###### Abstract.

Large-scale randomised experiments have become a standard tool for developing
products and improving user experience. To reduce losses from shipping harmful
changes experimental results are, in practice, often checked repeatedly, which
leads to inflated false alarm rates. To alleviate this problem, one can use
sequential testing techniques as they control false discovery rates despite
repeated checks. While multiple sequential testing methods exist in the
literature, they either restrict the number of interim checks the experimenter
can perform or have tuning parameters that require calibration. In this paper,
we propose a novel sequential testing method that does not limit the number of
interim checks and at the same time does not have any tuning parameters. The
proposed method is new and does not stem from existing experiment monitoring
procedures. It controls false discovery rates by “inverting” a bound on the
threshold crossing probability derived from a classical maximal inequality. We
demonstrate both in simulations and using real-world data that the proposed
method outperforms current state-of-the-art sequential tests for continuous
test monitoring. In addition, we illustrate the method’s effectiveness with a
real-world application on a major online fashion platform.

controlled experiments, randomised experiments, A/B testing, test monitoring

††journalyear: 2022††ccs: Mathematics of computing Hypothesis testing and
confidence interval computation††journalyear: 2024††copyright:
acmlicensed††conference: Proceedings of the 2024 ACM Conference; Date, 2024;
City, Country††booktitle: Proceedings of the 2024 ACM Conference††price:
XX.XX††doi: XXX††isbn: XXX

##  1\. Introduction

Online organizations, such as e-commerce platforms, increasingly utilize
experimentation approaches to develop products and improve the user
experience. While the ability to test changes and iterate on features fast is
associated with many benefits, it also substantially increases the risk of
exposing users to defective changes, degrading their experience. Unlike
traditional methods (such as the Student t-test) suffering from inflated false
detection (Type-I error) rates when applied repeatedly to interim experiment
results, sequential testing methods allow repeated checks for significance
without inflating the false discovery rate, making it possible to monitor
online experiments. In industrial applications, this is crucial as it allows
for the early termination of tests that have negative financial impacts,
leading to significant cost savings and faster iterations on improvements.

Sequential testing methods can be broadly divided into discrete and
continuous. Discrete sequential tests permit only a fixed number of interim
checks while continuous methods impose no such limit and allow checking for
significance at the experimenter’s will - even after each new observation. In
this paper, we mainly focus on continuous sequential tests. We find that the
absence of a restriction on the number of interim checks is useful in practice
as it avoids a situation where the experimenter and the stakeholders observe a
degradation in the metric of interest but have to wait until the next
checkpoint before assessing the statistical significance and making decisions.

An important practical challenge associated with currently available state-of-
the-art methods for continuous experiment monitoring (such as mSPRT and other
martingale methods; see (Howard et al., 2021; Lindon et al., 2022a; Ham et
al., 2023) and references therein) is the presence of tuning parameters that
need to be calibrated or superimposed.111For example, $\Phi$ in (Lindon et
al., 2022a, Section 3.4), $\rho$ in (Howard et al., 2021, Proposition 5), and
$\eta$ in (Ham et al., 2023, Section 4.2). Setting those parameters requires a
certain level of expertise that is not always available in applied teams. This
motivated us to develop an alternative sequential test which is simpler in
that it has no tuning parameters. All of its inputs are straightforward to
estimate from pre-experiment data. This new method for continuous test
monitoring is the subject of the paper.

Apart from our proposed method, we know only one continuous sequential test
that is rigorous (i.e. not a heuristic) and at the same time has no tuning
parameters. It is the procedure presented in (Miller, 2015). However, the
method from (Miller, 2015) only supports count-type metrics (such as the
number of clicks or purchases) whilst our proposed method can monitor metrics
of any type, including real-valued financial metrics such as revenue. We
emphasise that this is a non-trivial advancement as the theory presented in
(Miller, 2015) fundamentally supports only count-type metrics.

The proposed sequential testing method is novel and based on an alternative
theory relative to existing sequential tests.

Our work has both practical and theoretical contributions. On the practical
side, we design a sequential test that is flexible (supports unlimited interim
checks and arbitrary metrics) and easy to use (has no tuning parameters). At
the same time, from the theoretical perspective, this new test is justified by
a novel theory that cannot be viewed as an extension of existing frameworks.

We extensively validate our approach, showing both in simulations and using
real-world data that the proposed method outperforms existing state-of-the-art
continuous sequential tests. We share the associated code for greater
reproducibility (Kurennoy and Gurbanov, 2024).

To summarise, our main contributions are:

  * •

We propose a _flexible and easy-to-use_ sequential testing procedure. The
method supports unlimited interim checks and can monitor metrics of arbitrary
types (including real-valued financial metrics such as revenue). Contrary to
existing state-of-the-art continuous sequential tests, it has _no tuning
parameters_. All of its inputs are straightforward to estimate from pre-
experiment data.

  * •

We justify the proposed method with a _novel theory_ that is based on the
inversion of bounds obtained using (generalised) Levy’s inequalities (Loeve,
1960, Paragraph 29.1, A) and includes a false detection rate guarantee and a
sensitivity (power) statement.

  * •

We compare the proposed method with existing alternatives in an extensive
empirical study that includes simulation experiments and assessment based on
_real-world data_. We _share the associated code_ (Kurennoy and Gurbanov,
2024) to improve reproducibility and foster further research.

The remainder of this paper is structured as follows: Section 2 discusses
related work. Section 3 introduces the proposed method, explains its
derivation, and then presents its formal theory. Section 4 shows how our
method can be modified to handle unequal experiment groups or to distribute
its sensitivity across the duration of the experiment. In Section 5, we
investigate the performance of our sequential test in the framework of a
recent simulation study. In Section 6, we assess the correctness of the
proposed method using real-world data from a major online fashion platform.
Section 7 illustrates the effectiveness of the proposed method with a case
study from the fashion platform. Section 8 discusses the limitations of the
proposed approach and Section 9 concludes.

##  2\. Related Work

Our work is related to a large literature on sequential decision-making based
on decision rules defined via sequences of hypothesis tests, confidence
sequences, or sequences of ”always-valid” p-values. While the early work in
this field was primarily concerned with clinical trials and industrial quality
controls (Dodge and Romig, 1929; Bross, 1952; Armitage et al., 1954), more
recent work is typically motivated by the sequential nature of online
experiments, where users arrive sequentially and generate outcomes that are
observed fast relative to the duration of the experiment (Larsen et al., 2023;
Johari et al., 2022; Howard et al., 2021). Here, the goal of sequential
procedures is typically either to reduce the opportunity cost of longer
experiments, e.g. in the context of best-arm identification in multi-armed
bandit problems (Jamieson et al., 2013; Kaufmann et al., 2016; Yang et al.,
2017; Jamieson and Jain, 2018), or risk management, i.e. the quick detection
of harmful treatments (Lindon et al., 2022b; Ham et al., 2023). Our work is
primarily concerned with the latter and this section focuses on the two
dominant approaches underlying the monitoring methods used by large online
organizations (Schultzberg and Ankargren, 2023a).

###  2.1. Sequential Probability Ratio and Martingale Methods

Building on the seminal work of Wald (Wald, 1945) and Doob (Doob, 1953), most
modern sequential procedures are based on maximal inequalities for
appropriately constructed martingales that provide time-uniform statistical
guarantees. While Wald’s original sequential probability ratio test (SPRT) was
developed from scratch, its construction is essentially of this “martingale-
type”. The SPRT sequentially evaluates the likelihood ratio test statistic at
sample size $n$ for the null $H_{0}:\theta=\theta_{0}$ against the single
point alternative $H_{1}:\theta=\theta_{1}$ until either one of two bounds are
crossed and rejects or fails to reject the null accordingly. While the SPRT
terminates for finite $n$, the corresponding confidence sequence does not
shrink towards zero as $n\rightarrow\infty$ (Wald, 1947) and requires the
specification of a single point alternative, making it unsuitable for general
purpose monitoring of experiments. This shortcoming motivates likelihood ratio
based procedures that rely on the methods of mixtures, which replace the
single point alternative with a composite hypothesis and the likelihood ratio
with a mixture that is still a martingale under the null (Robbins, 1970).
These mixture sequential probability ratio tests (mSPRT) control false
discovery rates at any point in time but require the specification of a mixing
distribution. Methods to continuously monitor experiments based on the mSPRT,
as well as related but more involved methods constructed through other
martingales, are routinely used in large scale online organizations. (Johari
et al., 2022; Howard et al., 2021; Ham et al., 2023; Lindon and Malek, 2022;
Lindon et al., 2022a; Bibaut et al., 2022).

###  2.2. Group Sequential Methods

A conceptually different approach is followed by methods based on group
sequential tests (GST) (Pocock, 1977; O’Brien and Fleming, 1979; Slud and Wei,
1982; Gordon Lan and Demets, 1983). GST methods rely on (asymptotic)
characterizations of the joint distribution of a vector of test statistics
across increments of data, which permits the computation of sequential
boundaries that uniformly control the false discovery rate via multivariate
(numerical) integration. While GST methods also do not require to specify the
timing of tests in advance, they require the specification of a function
(”alpha spending function”) that governs the sensitivity across tests given a
pre-specified maximum sample size. As a consequence, GSTs, while typically
providing tighter confidence sequences than mSPRT-based methods, do not
strictly speaking support the continuous monitoring of tests.

##  3\. YEAST: YEt Another Sequential Test

In this section, we present a novel, alternative sequential testing method
proposed in this paper.

###  3.1. Problem Setup and Notation

Suppose we are conducting a randomised online experiment and have a certain
metric of interest. For ease of exposition let us assume that the experiment
has only two groups (control and treatment). We explain how to handle
experiments with multiple treatment groups in Section 4 below. We want to
constantly track the metric of interest and have a tool to check if the
difference between the groups is at any point large enough to conclude that
the treatment has an impact on the metric. The tool must have good control of
the false detection rate despite the repeated checks. In other words, it must
allow peeking into the interim results of the experiment without inflating the
false detection rate.

To develop a tool satisfying the above requirement, we consider the difference
$S$ in the total value of the metric of interest accumulated by the control
$(W=0)$ and the treatment $(W=1)$ groups. Typically, this difference $S$ can
be represented as a sum of increments over a stream of events. For example, if
the metric of interest is revenue then the tracked difference $S$ can be
decomposed into the sum of order values over checkout events where the order
value is taken with a positive sign if it comes from the control group and
with a negative sign if it comes from the treatment arm. Mathematically, the
increments can be written as

(1) $X_{i}=\left(1-2W_{i}\right)Y_{i},$

where $Y_{i}$ is the value or outcome associated with the $i$-th event (such
as the order value in the example above) and $W_{i}$ is the event assignment
indicator (it equals 0 if the event was generated by a subject from the
control group and 1 if its subject belongs to the treatment group). With this
notation at hand, the difference in the metric of interest after observing $n$
events can be written as the running sum

(2) $S_{n}=\sum_{i=1}^{n}X_{i},\quad n\geq 1,$

as illustrated in the example data set displayed in Table 1.

Table 1. An Example of Experiment Data i | timestamp | group | Y | X | S
---|---|---|---|---|---
1 | 2023-08-01 12:00:00 | control | 175.0 | 175.0 | 175.0
2 | 2023-08-01 12:00:02 | treatment | 35.5 | -35.5 | 139.5
3 | 2023-08-01 12:00:05 | treatment | 20.0 | -20.0 | 119.5
4 | 2023-08-01 12:00:10 | control | 100.0 | 100.0 | 219.5
… | … | … | … | … | …

So far we have used revenue monitoring as an example but one can think of
examples from other domains such as

  * •

video streaming (metric: hours streamed; the events are user sessions; the
event value is the total duration of videos streamed within the session),

  * •

online advertisement (metric: number of clicks; the events are clicks; the
event value is 1),

  * •

medical drug trial (metric: number of cases of side-effects; the events are
incoming patients; the event value is the indicator of side-effect presence
after taking the drug),

  * •

online subscription services (metric: number of new users who convert into a
paid subscription; the events are newly acquired users; the event value is the
indicator of subscribing),

  * •

banking (metric: total transaction fee; the events are transactions; the event
value is fee amount),

and others.

###  3.2. Assumptions and the Null Hypothesis

We will now state and discuss our assumptions and the null hypothesis.

Assumption 1 (Random Subject Assignment) The subjects are assigned to the
control or treatment uniformly at random and with equal probability.

We assume that the assignment is balanced only to derive a base version of our
proposed monitoring procedure. It is then straightforward to modify this base
version to handle unequal groups. We explain how to do this in Section 4
below.

We work with the following null hypothesis.

$\textup{H}_{0}$: the treatment does not affect either the event outcome
distribution or the frequency of events, so that

  * (a)

the event outcomes and the assignment indicators are independent from each
other, i. e.

(3)
$\\{Y_{1},\,Y_{2},\,\ldots\\}\perp\\!\\!\\!\perp\\{W_{1},\,W_{2},\,\ldots\\},$

  * (b)

and the frequency of events in each group is the same, implying (under
Assumption 1) that

(4) $\Pr\\{W_{i}=1\\}=0.5\quad\forall\,i\geq 1.$

In the setting of this paper, it is important to distinguish between subjects
(such as users) and events generated by subjects (such as orders). Throughout
the paper, we work with variables associated with events and not subjects. For
example, $Y_{i}$, $i\geq 1$, are event outcomes and $W_{i}$, $i\geq 1$, are
event assignment indicators. One implication of this is that even though the
subjects are assigned to one of the two groups independently at random, the
variables associated with individual events can be dependent (as some of them
can be generated by the same user). Therefore, in our main theory (Theorems 1
and 2), we allow the event outcomes $Y_{i}$, $i\geq 1$, to be dependent and,
likewise for the event assignment indicators $W_{i}$, $i\geq 1$. The part (a)
of the null hypothesis stated above only assumes that the two sets of
variables ($\\{Y_{1},\,Y_{2},\,\ldots\\}$ and $\\{W_{1},\,W_{2},\,\ldots\\}$)
are independent from each other but the variables within each of the two sets
are allowed to be dependent. Note that we also do not require $Y_{i}$, $i\geq
1$, to be identically distributed.

The null hypothesis stated above exhibits the difference between our setup and
the classical testing for a zero average treatment effect. Specifically,

  * •

our testing procedure acts on the level of events generated by experiment
subjects and not the experiment subjects themselves (for example, orders and
not customers);

  * •

we assume that under the null hypothesis, the treatment does not affect the
“event generation process” in any way.

In the revenue monitoring example, the latter implies that both the frequency
of orders and the order value distribution stay the same in both groups under
the null. We discuss the limitations of the considered setup in Section 8.

The approach described so far allows us to derive a simple but effective
sequential testing procedure we present next.

###  3.3. Proposed Sequential Testing Method

The proposed monitoring method consists of tracking the difference in the
metric of interest between the experiment groups and (continuously) comparing
it with a constant (alerting) boundary. Whenever the boundary is crossed, we
can conclude (with a predefined significance level $\alpha$) that the
treatment has an impact on the metric.

The procedure requires setting the maximum number of events to be observed
during the monitoring period that we denote by $N$. In other words, the
boundary is computed assuming that the monitoring will not continue after $N$
events have been collected. See Section 3.4 for how to set $N$ in practice.

The proposed method has only one other parameter that needs to be estimated
beforehand - the (scaled) variance of the difference in the metric of interest
at the end of monitoring, $V_{N}=var(S_{N})/N$. The estimator of $V_{N}$ needs
to be consistent under the null hypothesis. Given the estimate $\hat{V}_{N}$
(computed by such an estimator), the alerting boundary is set to

(5) $b^{*}=z_{1-\alpha/2}\cdot\sqrt{N\hat{V}_{N}},$

where $z_{1-\alpha/2}$ is the quantile of the standard normal distribution of
level $1-\alpha/2$. The proposed simple sequential test proceeds by tracking
$S_{n}$ and comparing it against $b^{*}$.

####  3.3.1. Informal Derivation of the Proposed Method

The derivation of the alerting boundary (5) follows two steps. First, we use
Levy’s inequality (Loeve, 1960, Paragraph 29.1, A) to bound the false
detection rate by the probability that the tracked sum exceeds the threshold
at the end of monitoring, i. e.

$FDR=\Pr\left\\{\max_{n=1}^{N}S_{n}>b\right\\}\leq
2\Pr\left\\{S_{N}>b\right\\}.$

In the second step we use the Central Limit Theorem to approximate the latter
probability,

$\Pr\left\\{S_{N}>b\right\\}\approx
1-\Phi\left(\frac{b}{\sqrt{var(S_{N})}}\right)=1-\Phi\left(\frac{b}{\sqrt{NV_{N}}}\right),$

where $\Phi$ denotes the CDF of the standard normal distribution. Then by
setting the threshold $b$ to $z_{1-\alpha/2}\cdot\sqrt{NV_{N}}$ (cf. (5)), we
ensure that the false detection rate is under control, i. e.
$FDR\lessapprox\alpha$. This is formalised in Theorem 1 below.

The proposed testing procedure is defined in Algorithm 1 and will be referred
to as _YEAST_ (from YEt Another Sequential Test). The estimation of $V_{N}$ is
discussed in Section 3.4 below.

Algorithm 1 YEAST

$N$, $\hat{V}_{N}$

$b^{*}\leftarrow z_{1-\alpha/2}\sqrt{N\hat{V}_{N}}$

for $n=1,\,\ldots,\,N$ do

if $S_{n}>b^{*}$ then

flag significance

end if

end for

For a two-sided test, we track $|S_{n}|$ and compare it against

(6) $b^{*}_{2-sided}=z_{1-\alpha/4}\cdot\sqrt{N\hat{V}_{N}},$

where $z_{1-\alpha/2}$ is the quantile of the standard normal distribution of
level $1-\alpha/4$. Mind that differently from the non-sequential testing, the
significance level $\alpha$ needs to be divided by 2 for the one-sided test
and by 4 for the two-sided test. This is due to the factor of two in the
right-hand side of inequalities (14) and (15) below.

###  3.4. Setting the Input Parameters

As was described in the previous subsection, the computation of the alerting
boundary of the proposed sequential testing procedure requires two inputs: the
number of events $N$ to be observed during the monitoring and an estimate
$\hat{V}_{N}$ of the (scaled) variance of the tracked metric difference at the
end of the monitoring. In this subsection, we discuss how to provide those
inputs in practice.

To set $N$, we estimate the expected number of events to be collected over the
experiment time frame. In the simplest form, this can amount to setting $N$ to
the number of events observed in a pre-experiment period of the same duration
as the experiment we want to monitor.

For variance estimation, we suggest using pre-experiment data as well.
Obtaining an accurate estimate this way requires the pre-experiment period to
be representative of the experiment time. However, in cases where this cannot
be assumed, the validity of fixed-duration randomised experiments is
questionable in principle (as the effects measured during such experiments may
not generalise beyond their time frames). So in the context of A/B testing,
this is a natural assumption.

As was noted in Section 3.2, the event outcomes $Y_{i}$, $i\geq 1$, can be
dependent because some of the events are generated by the same user. The same
holds for the event assignment indicators $W_{i}$, $i\geq 1$. Consequently,
the increments $X_{i}$, $i\geq 1$, of the monitored trajectory can be serially
correlated in which case the variance $var(S_{N})$ does not equal the sum of
variances $var(X_{i})$. The serial correlation of the increments has a
clustered nature in this case because variables corresponding to different
users are independent due to random assignment. Therefore, we propose to use
an estimator that is robust to clustered serial correlation (A. Colin Cameron
and Miller, 2011) to estimate $V_{N}$. Implementations of such estimators are
readily available, for example as part of the sandwich package in R (Zeileis
et al., 2020; Zeileis, 2004) (see function vcovCL). The cluster variable in
this case is the user identifier. As we will show in our experiments with
real-world data in Section 6, the use of a robust (clustered) variance
estimator (instead of the simple sum of $var(X_{i})$) can be utterly important
for effective control of the false detection rate.

Note that the inputs of the proposed method ($N$ and $V_{N}$) can be directly
estimated from pre-experiment data and setting them does not require tuning or
leveraging expert knowledge. The absence of tuning parameters makes the
proposed Algorithm 1 (YEAST) easy to use.

###  3.5. Formal Statements

In this subsection, we will justify the correctness of the proposed testing
procedure. This includes a statement demonstrating that the proposed method
controls the false detection rate (Theorem 1), and a statement showing that
the power of the method is asymptotically one (Theorem 2).

In the formal statements below, $(\Omega,\,\mathcal{F},\,\Pr)$ is a
probability space and $\mathrm{I}\\!\mathrm{R}$ stands for the set of real
numbers. As before, $\Phi$ denotes the CDF of the standard normal
distribution. The proofs of the statements can be found in Appendix A.

The first theorem (see below) states that when the treatment has no actual
impact, detections occur sufficiently rarely. Its proof (see Appendix A) has
two steps. First, we bound the chance of “crossing the threshold” at any
moment during the monitoring by twice the probability of crossing it at the
last moment. Secondly, we approximate the latter probability with the help of
the Central Limit Theorem (noting that the tracked difference at the end of
monitoring is the sum of a large number of increments). This allows us to
construct the threshold appropriately, ensuring that the method’s false
detection rate is small enough.

###### Theorem 1 (False Detection Rate Control).

Let $Y_{i}\colon\Omega\mapsto\mathrm{I}\\!\mathrm{R}$ and
$W_{i}\colon\Omega\mapsto\\{0,\,1\\}$, $i=1,\,\ldots,\,N$, be random variables
and let $S_{n}$, $n\geq 1$, be the running sum defined by (1)–(2). Suppose
that Assumption 1 and the null hypothesis (3)–(4) hold and that a version of
the Central Limit Theorem is valid for the sequence of random variables
$X_{1},\,X_{2},\,\ldots$ defined by (1). Assume that $var(S_{N})>0$ for all
$N\geq 1$ and set $V_{N}:=var(S_{N})/N$.

Then for any $\alpha\in[0,\,1]$ and any $\varepsilon>0$ there exists
$N_{\alpha,\,\varepsilon}$ such that for all $N\geq N_{\alpha,\,\varepsilon}$
we have

$\Pr\left\\{\max_{n=1}^{N}S_{n}>z_{1-\alpha/2}\sqrt{NV_{N}}\right\\}\leq\alpha+\varepsilon,$

where $z_{1-\alpha/2}$ is the quantile of the standard normal distribution of
level $1-\alpha/2$.

Note that we deliberately do not spell specific conditions ensuring that the
central limit properties hold for the sequence of increments $\\{X_{i}\\}$.
Multiple sets of such conditions exist. A good overview of them can be found
in (White, 1984, Chapter V).

Our second theorem claims that when the treatment has an effect, its detection
can be made as certain as desired if we monitor for sufficiently long. The
idea of its proof is to note that the probability of detecting the effect at
any moment during the monitoring is at least as high as the probability of
detecting it at the very end. Then we leverage the Central Limit Theorem to
show that (when the treatment effect exists) the latter probability approaches
1 as the monitoring duration increases, completing the proof (see Appendix A
for details).

###### Theorem 2 (Power).

Let $Y_{i}\colon\Omega\mapsto\mathrm{I}\\!\mathrm{R}$ and
$W_{i}\colon\Omega\mapsto\\{0,\,1\\}$, $i=1,\,\ldots,\,N$, be random
variables, and let $S_{n}$, $n\geq 1$, be the running sum defined by (1)–(2).
Suppose that Assumption 1 holds and a version of the Central Limit Theorem is
valid for the sequence of random variables $X_{1},\,X_{2},\,\ldots$ defined by
(1). Assume that $var(S_{N})>0$ for all $N\geq 1$ and that the set of
variables $V_{N}=var(S_{N})/N$, $N\geq 1$, is stochastically bounded. Finally,
let $E[X_{i}]=\mu_{i}>0$ (implying that the null hypothesis (3)–(4) is
violated).

Then

$\Pr\left\\{\max_{n=1}^{N}S_{n}>z_{1-\alpha/2}\sqrt{NV_{N}}\right\\}\to
1\quad\textup{as}\quad N\to\infty.$

Similar FDR control and “power” theorems hold for the two-sided version of our
method and can be obtained in full analogy to the above statements.

##  4\. Extensions

###  4.1. Unequal Groups And Multiple Treatment Groups

For randomised experiments with unequal groups, one can down-sample the larger
group when computing the cumulative KPI difference $S_{n}$ and then apply the
monitoring procedure (Algorithm 1). It is important to note that the down-
sampling must be done with respect to subjects and not events (e. g. following
the e-commerce example, on the level of users, not orders). The reason for
that is that the bigger group can be causing harm by making subjects generate
fewer events on average, and we want the monitoring to be sensitive to such
situations.

If there are multiple treatment groups, one can monitor each treatment
separately applying the procedure developed in Section 3 to each treatment-
control pair (with an appropriately reduced significance level to correct for
multiple comparisons). On the other hand, if the purpose of monitoring is to
stop early when the experiment as a whole is causing harm, we recommend
handling all of the treatment groups collectively (as if it was a single
group) and then down-sampling it as in the case of unequal groups.

We also considered reweighting instead of downsampling (based on the known
ratio between the group sizes). Yet, extending the theory to the weighted sum
case turns out to be challenging for now. The reason is that reweighting sets
the mean of the sum to zero but is not guaranteed to make its median zero as
needed to bound the crossing probability in our case. We plan to revisit this
in our future work.

###  4.2. Progressive Thresholds

The method proposed in Section 3 gives a constant significance boundary.
Graphically, it corresponds to a horizontal line (see Figure 2 below for an
example). If at any moment during the experiment, the cumulative difference in
the metric of interest between the control and treatment (i.e. the observed
loss) crosses this line, we consider it as evidence that the treatment is
causing harm. As we will show in this section, a similar idea can be used to
produce a sequence of significance thresholds, increasing in magnitude. This
way, we can maintain a tighter ”limit” on the observed loss in the early days
of the experiment and guard against particularly harmful treatments more
effectively. Such a sequence of thresholds visually forms a staircase instead
of a line (Figure 2).

_Only_ in this section, we will maintain one extra assumption (in addition to
Assumption 1 stated in Section 3.2).

Assumption 2 (Independence of Increments) The increments
$X_{i}=(1-2W_{i})Y_{i}$, $i\geq 1$, are independent.

This assumption generally holds when all subjects generate only one event as
in the examples of a drug trial and online subscription services mentioned at
the end of Section 3.1 or when the random assignment is done on the event
level (e. g. when the events are sessions and we randomise per session).

To obtain progressive thresholds, let us divide the experiment timeline into
$K$ periods (e.g. days) and suppose we want to set $K$ alerting thresholds
$b_{1},\,\ldots,\,b_{K}$. Let $N_{k}$ be the sample size (number of events) in
period $k$. Furthermore, let $S_{n}$ be the difference in the total metric
value between the control and treatment as in (2). Finally, suppose that
$M_{k}$ is the maximum value of $S_{n}$ in period $k$. With this notation at
hand, we can write the false detection probability as follows.

(7) $FDR=\Pr\\{M_{1}>b_{1}\\}+\Pr\\{M_{1}\leq b_{1},M_{2}>b_{2}\\}+{}\\\
{}+\ldots+\Pr\\{M_{1}\leq b_{1},\,\ldots,\,M_{K-1}\leq
b_{K-1},\,M_{K}>b_{K}\\}.$

According to the theory from Section 3 (see Lemma 2), the first term in the
above sum can be bounded by $2\Pr\\{S_{N_{1}}\geq b_{1}\\}$ which, under the
null hypothesis, can be approximated with
$2\left(1-\Phi\left(\frac{b_{1}}{\sqrt{U_{1}}}\right)\right)$, where $U_{1}$
denotes $var(S_{N_{1}})$ and, as before, $\Phi$ stands for the CDF of the
standard normal distribution. Let us now consider the remaining terms.

For any $k=1,\,\ldots,\,K$ we can write

$\Pr\\{M_{1}\leq b_{1},\,\ldots,\,M_{k-1}\leq b_{k-1},\,M_{k}>b_{k}\\}\\\
\leq\Pr\\{S_{N_{1}+\,\ldots\,+N_{k-1}}\leq
b_{k-1},M_{k}>b_{k}\\}\stackrel{{\scriptstyle Assumption~{}2}}{{=}}\\\
=\int_{-\infty}^{b_{k-1}}\Bigg{[}\Pr\left\\{\max_{\ell=1,\,\ldots
N_{k}}\sum_{i=1}^{\ell}X_{N_{1}+\ldots+N_{k-1}+i}\geq b_{k}-x\right\\}\\\
\times f_{S_{N_{1}+\ldots+N_{k-1}}}(x)\Bigg{]}dx,$

where $f_{S_{N_{1}+\ldots+N_{k-1}}}$ is the density of
$S_{N_{1}+\,\ldots\,+N_{k-1}}$. As in Theorem 1, we assume that a Central
Limit Theorem holds for the sequence $\\{X_{i}\\}$. Then, under the null
hypothesis, $f_{S_{N_{1}+\ldots+N_{k-1}}}$ can be approximated by the density
of a centred normal variable with variance
$U_{k-1}=var(S_{N_{1}+\,\ldots\,+N_{k-1}})$. At the same time, by applying
Lemma 2, we have

$\Pr\left\\{\max_{\ell=1,\,\ldots
N_{k}}\sum_{i=1}^{\ell}X_{N_{1}+\ldots+N_{k-1}+i}>b_{k}-x\right\\}\\\ \leq
2\Pr\left\\{\sum_{i=1}^{N_{k}}X_{N_{1}+\ldots+N_{k-1}+i}\geq
b_{k}-x\right\\}\approx
2\left(1-\Phi\left(\frac{b_{k}-x}{\sqrt{u_{k}}}\right)\right),$

where $u_{k}$ denotes the variance of
$\sum_{i=1}^{N_{k}}X_{N_{1}+\ldots+N_{k-1}+i}$. Therefore, $P\\{M_{1}\leq
b_{1},\,\ldots,\,M_{k-1}\leq b_{k-1},\,M_{k}>b_{k}\\}$ can be (approximately)
bounded by

$I_{k}=2\int_{-\infty}^{b_{k-1}}\Bigg{[}\left(1-\Phi\left(\frac{b_{k}-x}{\sqrt{u_{k}}}\right)\right)\varphi\left(\frac{x}{\sqrt{U_{k-1}}}\right)\left(\sqrt{U_{k-1}}\right)^{-1}\Bigg{]}dx,$

where $\varphi$ denotes the density of a standard normal random variable.
Setting $Z_{k}=\Phi\left(\frac{b_{k-1}}{\sqrt{U_{k-1}}}\right)$, we can
further rewrite it like this:

(8) $I_{k}=2Z_{k}(1-J_{k}),$

where

(9)
$J_{k}=\int_{-\infty}^{b_{k-1}}\Bigg{[}\Phi\left(\frac{b_{k}-x}{\sqrt{u_{k}}}\right)\varphi\left(\frac{x}{\sqrt{U_{k-1}}}\right)\left(Z_{k}\sqrt{U_{k-1}}\right)^{-1}\Bigg{]}dx.$

Note that the integral $J_{k}$ is a convolution between a centred normal and a
truncated normal distribution (evaluated at $b_{k}$). To compute this integral
one can apply methods listed in (Drysdale, 2021), for example, (Kim, 2006,
Formula 2.4).

In summary, from (7)–(9), we have that for any given set of thresholds
$b_{1},\,\ldots,\,b_{K}$, the associated false detection rate can be
(approximately) bounded by

$FDRBound=2\left(1-\Phi\left(\frac{b_{1}}{\sqrt{U_{1}}}\right)+\sum_{k=2}^{K}Z_{k}(1-J_{k})\right).$

Having the (approximate) FDR bound (4.2), one can find a set of thresholds
such that the bound is below the desired significance level $\alpha$. For
example, the thresholds can be initialised with
$b^{0}_{k}=z_{1-\alpha/2}\cdot\sqrt{\hat{U}_{k}}$, $k=1,\,\ldots,\,K$, where
$\hat{U}_{k}$ is an estimate of $var(S_{N_{1}+\ldots+N_{k}})$ and then
multiplied by $(1+\varepsilon)$ (where $\varepsilon$ is a chosen small enough
number) until the FDR bound (4.2) gets below $\alpha$. This procedure is
formally described in Algorithm 2.

Algorithm 2 “Staircase” Significance Boundary Computation

$K$, $N_{1},\,\ldots,\,N_{K}$, $\hat{U}_{1},\,\ldots,\,\hat{U}_{K}$,
$\hat{u}_{1},\,\ldots,\,\hat{u}_{K}$, $\varepsilon>0$

$b_{1}^{*},\,\ldots,\,b_{K}^{*}\colon
FDRBound(b_{1}^{*},\,\ldots,\,b_{K}^{*})\leq\alpha$

for k=1, …, K do

$b_{k}^{*}\leftarrow z_{1-\alpha/2}\sqrt{\hat{U}_{k}}$

end for

while $FDRBound(b_{1}^{*},\,\ldots,\,b_{K}^{*})>\alpha$ do

for $k=1,\,\ldots,\,K$ do

$b_{k}^{*}\leftarrow(1+\varepsilon)b_{k}^{*}$

end for

end while

As was mentioned before, maintaining a sequence of thresholds allows having a
tighter limit on the observed loss in the early days of the experiment. This
makes the monitoring more effective at reducing risks associated with
particularly harmful treatments but comes at the cost of having a looser
significance boundary closer to the end of the test.

The procedure developed in this section resembles the so-called alpha-spending
approach (Pocock, 1977; O’Brien and Fleming, 1979; Slud and Wei, 1982; Gordon
Lan and Demets, 1983) but, importantly, allows for continuous monitoring
rather than a discrete number of checks. In what follows, we will refer to it
as _pYEAST_.

##  5\. Simulation Experiments

When designing our simulation experiments, we sought to extend an existing
setup that has already been reviewed by the relevant community of researchers
and practitioners. Hence, our simulation setting follows a recent study
(Schultzberg and Ankargren, 2023a) and the implementation is based on the
associated code (Schultzberg and Ankargren, 2023b) which we extended with the
proposed methods from Sections 3 and 4.2 above.

In particular, as in (Schultzberg and Ankargren, 2023b), we generate $N$
observations from control, $Y_{i}^{c}$, $i=1,\,\ldots,\,N$, and $N$
observations from treatment, $Y_{i}^{t}$, with $N$ set to 500. Both
$Y_{i}^{c}$ and $Y_{i}^{t}$ are generated as IID random variables. The effect
size (on the mean) took values 0.0, 0.1, 0.2, 0.3, and 0.4. The $X_{i}$
variables (increments in the metric difference between the two groups) were
computed as $X_{i}=Y_{i}^{c}-Y_{i}^{t}$, $i=1,\,\ldots,\,N$.

We conducted two simulation experiments. The first (main) experiment repeats
the simulation study from (Schultzberg and Ankargren, 2023a) but adds the
methods proposed in Section 3 and Section 4.2 to the evaluation. In the second
set of simulations, we explored the behaviour of the compared methods when the
increments are non-normal.

In all of the experiments, the target significance level was set to 5% and the
number of replications was set to 100,000.

The simulation code for this paper can be found in the associated git
repository (Kurennoy and Gurbanov, 2024).

###  5.1. Main Simulation Experiment

In this experiment, the observations $Y_{i}^{c}$ and $Y_{i}^{t}$ were drawn
from normal distributions with parameters $(1,\,1)$ and $(1+\xi,\,1)$,
respectively. The effect size $\xi$ took values 0.0, 0.1, 0.2, 0.3, and 0.4.
The seed of the random generator was set to 8163 (as in (Schultzberg and
Ankargren, 2023a, b)).

Below we present the list of methods we compared.

YEAST The simple sequential method using a constant significance boundary (5)
as proposed in Section 3.

pYEAST The method using a staircase significance boundary as proposed in
Section 4.2 (see Algorithm 2). We set $K$ (number of periods) to 7 and 14 and
denote the respective instances of the method by pYEAST7 and pYEAST14.

mSPRT The mixture sequential probability ratio test (Robbins, 1970; Lindon et
al., 2022b). We set the tuning parameter of the method to 11, 25, and 100 and
denote the corresponding versions as mSRTphi11, mSRTphi25, and mSRTphi50.

GAVI The generalization of the always valid inference, as proposed in (Howard
et al., 2021). As in (Schultzberg and Ankargren, 2023a), we set the numerator
of parameter $\rho$ of the method to 250, 500, and 750 and denote the
corresponding instances of the method by GAVI250, GAVI500, and GAVI750,
respectively.

CAA The corrected-alpha approach (STATSIG, 2023).

Bonferroni A naive approach using Bonferroni corrections.

All of the compared methods were employed in the continuous monitoring mode
meaning that the check for significance was performed after each observation.
In Appendix B we report additional evaluation results for the case where the
methods are used in the ”discrete mode” (i. e. with a fixed number of interim
significance checks).

For each experimental setting, we conducted 100,000 replications. Each
replication can result in a detection or no-detection. A detection occurs when
the respective test flags significance (at any point of the monitoring
process). We compute the share of replications where a detection occurs. When
the treatment does not have an effect this share is the so-called (empirical)
false detection rate (or, synonymously, false positive rate, type-I error, or
test “size”). In settings where the treatment has an effect, this share is the
(empirical) power. We report the measured false detection rate and power in
Table 2. Note that we do not report power for those methods that have an
inflated false detection rate. Otherwise, the comparison would not be fair
because one can generally obtain extra power by inflating the false detection
rate. We also omit the column corresponding to the effect size of 0.4 because
all methods under comparison had a power very close to one in that case. The
method with the highest power is shown in bold.

One can see from the table that the procedure proposed in Section 3 (Algorithm
1, YEAST) demonstrated the highest power among the continuous monitoring
approaches that do not inflate the false detection rate. It means that while
it kept the false positive rate under control when there was no treatment
effect, YEAST had the highest sensitivity in simulations where the treatment
had an effect on the metric of interest. The “staircase” extension of the
proposed method developed in Section 4.2 (pYEAST7 and pYEAST14) came as a
runner-up: it also kept the false positive rate below the prespecified level
and had the second and third highest power (after YEAST).

Table 2. Simulation Experiment:
False Detection Rate and Power | effect size | 0.0 | 0.1 | 0.2 | 0.3
---|---|---|---|---|---
| method |  |  |  |
1 | YEAST | 0.05 | 0.44 | 0.92 | 1.00
2 | pYEAST7 | 0.03 | 0.30 | 0.82 | 0.99
3 | pYEAST14 | 0.03 | 0.25 | 0.78 | 0.99
4 | mSPRTphi100 | 0.02 | 0.22 | 0.76 | 0.99
5 | mSPRTphi11 | 0.03 | 0.22 | 0.73 | 0.98
6 | mSPRTphi25 | 0.03 | 0.24 | 0.76 | 0.99
7 | GAVI250 | 0.02 | 0.24 | 0.76 | 0.99
8 | GAVI500 | 0.02 | 0.23 | 0.76 | 0.99
9 | GAVI750 | 0.01 | 0.21 | 0.75 | 0.99
10 | CAA | 0.07 | - | - | -
11 | Bonferroni | 0.00 | 0.04 | 0.40 | 0.90

The main benefit of sequential testing is the ability to stop the experiment
early (once significance is flagged upon one of the interim checks). This
allows saving time and, in case the treatment is harmful, reduce financial
losses. Thus, the amount of savings that is generated by a sequential test on
average is another important metric and we report it in Table 3. The savings
are measured as follows: if a method identified the effect after the arrival
of 10% of the total number of observations, the associated sample (or,
equivalently, time) savings would be 90%.

Among the compared methods, a variation of our proposed method with
progressive thresholds, pYEAST7, generated the largest savings on average for
most of the effect sizes we experimented with. Overall, pYEAST generated
higher savings than YEAST, which is expected since the purpose of having a
staircase boundary instead of a constant significance threshold was to be able
to detect large treatment effects faster. Note that pYEAST14 performed very
closely to pYEAST7 in terms of the generated savings, so the results were not
sensitive to the choice of the number of periods (or boundary “stairs”, $K$)
in the pYEAST method.

Table 3. Simulation Experiment:
Sample/Time Savings | effect size | 0.1 | 0.2 | 0.3 | 0.4
---|---|---|---|---|---
| method |  |  |  |
1 | YEAST | 0.13 | 0.39 | 0.58 | 0.69
2 | pYEAST7 | 0.14 | 0.44 | 0.69 | 0.80
3 | pYEAST14 | 0.12 | 0.41 | 0.68 | 0.80
4 | mSPRTphi100 | 0.09 | 0.35 | 0.62 | 0.75
5 | mSPRTphi11 | 0.12 | 0.41 | 0.68 | 0.82
6 | mSPRTphi25 | 0.12 | 0.41 | 0.68 | 0.81
7 | GAVI250 | 0.12 | 0.40 | 0.67 | 0.80
8 | GAVI500 | 0.10 | 0.37 | 0.63 | 0.77
9 | GAVI750 | 0.08 | 0.34 | 0.60 | 0.74
10 | Bonferroni | 0.02 | 0.19 | 0.45 | 0.67

###  5.2. Non-Normal Data

The derivation of the methods proposed in Section 3 and 4.2 involves the
application of the Central Limit Theorem to sums of $X_{i}$ (increments in the
metric difference between the two groups). For a fixed sample size, the
quality of the normal approximation depends on the distribution of $X_{i}$. In
this section, we explore the performance of the sequential testing methods
proposed in Sections 3 and 4.2 (YEAST, pYEAST7, and pYEAST14) in situations
where the distribution of $Y_{i}^{c}$ and $Y_{i}^{t}$ is not normal.
Specifically, we used two alternative distributions: Student’s t which has
heavier tales than the normal distribution and Gamma which is asymmetric.
Student’s t distribution had 3 degrees of freedom and was shifted by
$\sqrt{3}$ for the control and by $\sqrt{3}(1+\xi)$ for the treatment. The
shifting was done to maintain the same coefficient of variation as in Section
5.1). The Gamma distribution had its shape parameter set to 1.0. The scale
parameter equaled $2$ for the control and $2(1+\xi)$ for the treatment. The
effect size $\xi$ took the same values as in the main experiment:
$\xi=0.0,\,0.1,\,0.2,\,0.3,\,0.4$. The random seed was set to 2023 for the
simulations with the Student’s t distribution and to 2024 for the simulations
with the Gamma distribution. The two seeds were different to avoid dependence
across the two sets of simulations.

The results are depicted in Figure 1. The first data point on each line
corresponds to the case of no treatment effect and therefore represets the
false detection rate. The remaining points represent the power for different
treatment effect sizes. Similarly to the normal case, the CAA method was the
only one with an inflated false detection rate (and we omit it from the plot
for that reason). Among the methods that did not have an inflated false
positive rate, the method proposed in Section 3, YEAST, performed best with a
considerably higher power curve (both for the Gamma and the Student’s t
distribution cases).

Figure 1. Power Curves (under non-normal increment distributions

##  6\. Validation Using Real-World Data

We further validated our proposed approach using real data from a major
e-commerce platform. The data consisted of checkout events (placed orders)
from 22 weeks. It was split into periods of two weeks, which is a typical
A/B-experiment duration on the e-commerce platform. We used all but the first
two-week periods for validation.

This data naturally contained outliers (such as resellers that have a huge
purchase volume and can easily skew the monitored metric difference in favour
of one of the groups). To remove them, we applied revenue capping at the
99.9th percentile of the total revenue generated by a customer. For capping in
each validation period, the percentile was computed using the preceding two-
week period. The capping was applied progressively as follows: a given
customer was only monitored while his/her revenue stayed within the cap and
all subsequent events _from that customer_ were discarded.

For each validation period, we randomly assigned users to control and
treatment 10,000 times222The random seed for replicating the assignments was
set to $2024+i$, where $i$ is the number of the respective two-week validation
period, $i=1,\,\ldots,\,10$., computed the cumulative difference in the
revenue between control and treatment for each assignment replication, applied
the sequential tests to monitor the cumulative difference, and measured the
frequency of detections over the replications. We then pooled the measurements
from the 10 validation periods and computed the detection rate from the
resulting 100,000 data points. Since no actual treatment was applied to the
replicated treatment groups, the null hypothesis was true, and the computed
rates were false detection rates. They are reported in Table 4 below. In all
cases, the width of the confidence interval for the false detection rate was
an order of magnitude smaller than the false detection rate itself and hence
we omitted the confidence intervals from the table so as not to clutter it.

When applying the sequential tests to the data from a given period, the
estimation of input parameters was conducted on the preceding two-week period.
As was mentioned in Section 3.1, we employed the sequential tests with two
different variance estimates. One of the two assumed that the increments
$X_{i}$ of the monitored difference-in-sum are independent. The other was an
estimator robust to cluster serial correlation, that is - to the correlation
between increments corresponding to the same user in our case. The results
corresponding to the use of the two estimators are reported in columns ‘non-
robust‘ and ‘robust‘ of Table 4, respectively.333We used the vcovCL function
of the sandwich R package to compute the robust variance estimate. To reduce
the sparsity in the time dimension, we summed the data (revenue increments) by
user-hour before computing the estimator. We also used downsampling (in the
user dimension) to speed up the computation of the estimate.

Two observations from our experiment with the real-world data are in order.
First, the proposed method (YEAST) demonstrated the best result among the
compared methods (i. e. its false detection rate was closest to the nominal
significance level) when using the robust variance estimator. Second, all the
methods had a considerably inflated false detection rate when using the non-
robust estimator. The latter highlights the importance of accounting for the
correlation in the events generated by the same user when estimating the
variance.

The code implementing our experiment with the real-world data is openly
available in the git repository of the paper (Kurennoy and Gurbanov, 2024).

Table 4. Validation Using Real-World Data: False Detection Rate  | variance estimate | robust | non-robust
---|---|---|---
| method |  |
1 | YEAST | 0.04 | 0.12
2 | pYEAST14 | 0.02 | 0.09
3 | mSPRT100 | 0.03 | 0.16
4 | mSPRT011 | 0.02 | 0.13
5 | mSPRT025 | 0.02 | 0.14
6 | GAVI250 | 0.02 | 0.15
7 | GAVI500 | 0.03 | 0.16
8 | GAVI750 | 0.03 | 0.16
9 | CAA | 0.07 | 0.15

##  7\. Real-World Application

The proposed sequential testing method was implemented by a major ecommerce
platform and have been used there as a standard tool to monitor A/B tests in
production. We picked two of those tests as an illustration and will present
them in this section.

When monitoring live online experiments, we use the progressive variant of the
method (pSST) with the number of periods set to the number of days over which
the experiment is to run. Our main goal is to mitigate the risk of financial
losses and therefore we use the one-sided test (to increase the power at
detecting performance degradations). The significance level is set to 5%.

The trajectory of the difference in the Gross Merchandise Value (GMV) between
treatment and control in the two examples is shown in Figure 2 (blue solid
line). The red line shows the pSST thresholds (i.e. the alerting boundary). We
also plot the SST boundary using a dashed red line for comparison.

Both of the two experiments were scheduled to run for 2 weeks. In the first
experiment (see the upper plot), the monitoring detected a performance
degradation on day 4 and the test was stopped early, which allowed us to save
revenue and iterate faster. The lower plot corresponds to the second
experiment in which no detection occurred. The plot shows only the first few
days of the test but it was run for the full duration of 14 days and its
treatment improved engagement metrics without harming the financials (as
concluded by the post-analysis). The second experiment is particularly
interesting as its GMV trajectory was tracking below zero close to the
beginning of the test. It showcases the importance of using rigorous
monitoring procedures as naive monitoring could have raised concerns over the
observed loss at the start of the test and could potentially lead to the test
being stopped and unfairly deemed unsuccessful. In this example, the use of
our monitoring method helped avoid concerns and make the right decision to
continue the test.

Figure 2. Monitoring Examples

##  8\. Limitations

We have seen in Sections 5 and 6 that despite its simplicity, the proposed
monitoring procedure is very effective at controlling the type-I error and has
a high power. Yet there are scenarios where the underlying null hypothesis of
no effect on the data generation process is too strong, potentially leading to
some “undesirable sensitivity”. In other words, sometimes the treatment can
alter the data generation process but have a neutral average effect on the
metric of interest. We can think of (at least) two such scenarios.

Situations where the treatment brings multiple changes to the data generation
process and those changes counter-balance each other leading to a neutral
average effect on the metric. For example, the treatment can make users place
fewer orders but increase the average order value at the same time so the two
effects neutralise each other and the revenue per user stays intact.

Situations where the treatment changes the variance of the event outcome,
leaving the average unaffected. For example, the treatment can increase the
variability of the order values without changing their average.

If in a given practical application, having sensitivity to one of the
scenarios above is highly undesirable, it is safer to adhere to alternative
sequential testing methods (e.g. (Howard et al., 2021; Ham et al., 2023)). We
consider extending the proposed method to cover situations of those types as a
topic for future research.

Note that none of the problematic two scenarios can materialise when the event
outcome is constant (i. e. the metric of interest simply counts events). In
addition, the first scenario cannot hold when the treatment cannot affect the
number of events, for example, when the events correspond to incoming patients
in a medical study.

From a theoretical perspective, the amount of oversensitivity of the proposed
method in the above two situations would depend on how much the median of the
“residual sum” (or the sum of the remaining increments until the end of
monitoring) deviates from zero. Intuitively, in the early steps of monitoring,
this should not be a problem because the number of remaining increments is
large and their sum’s distribution would be approximately symmetric due to the
Central Limit Theorem. Yet, closer to the end of monitoring, this deviation
can be considerable. Mitigation strategies can include stopping the monitoring
early or setting the threshold as if the monitoring is to be run for longer
than in reality. The effectiveness of those mitigation strategies and the
tradeoff with the power would depend on how quickly the distribution of
increment sums converges to a normal distribution. Overall, we see studying
the behaviour of our proposed method in situations mentioned in this section
and seeking mitigation strategies as a large topic for future research.

It is worth emphasising that in some situations it can be desirable to capture
all relevant potential defects (beyond the effect on the mean) (Lindon et al.,
2022b). For example, an increased outcome variance can indicate that the
effect of the treatment is heterogeneous and some important subgroups of the
population can be negatively affected (even though the overall mean stays the
same). In such scenarios, the extra sensitivity mentioned in this section is
not really a limitation but an advantage instead.

##  9\. Conclusion

Motivated by practical challenges that undermine the adoption of sequential
testing procedures, this paper proposes a simple yet powerful method for
constructing confidence boundaries that permit continuous monitoring of online
experiments - both for discrete and continuous variables. It has no tuning
parameters and all of its inputs can be straightforwardly estimated from pre-
experiment data, which makes it easy to adopt. The method’s effectiveness is
demonstrated both in simulations and using real-world data. We hope the
proposed method will enable wider use of sequential testing in practice,
increasing the speed of experimentation and lowering its cost.

## References

  * (1)
  * A. Colin Cameron and Miller (2011) Jonah B. Gelbach A. Colin Cameron and Douglas L. Miller. 2011.  Robust Inference With Multiway Clustering.  _Journal of Business & Economic Statistics_ 29, 2 (2011), 238–249.  https://doi.org/10.1198/jbes.2010.07136 arXiv:https://doi.org/10.1198/jbes.2010.07136
  * Armitage et al. (1954) Peter Armitage et al. 1954\.  Sequential tests in prophylactic and therapeutic trials.  _Quarterly Journal of Medicine_ 23, 91 (1954), 255–74.
  * Bibaut et al. (2022) Aurélien F. Bibaut, Nathan Kallus, and Michael Lindon. 2022.  Near-Optimal Non-Parametric Sequential Tests and Confidence Sequences with Possibly Dependent Observations.  https://api.semanticscholar.org/CorpusID:255341081
  * Bross (1952) I Bross. 1952.  Sequential medical plans.  _Biometrics_ 8, 3 (1952), 188–205.
  * Dodge and Romig (1929) Harold French Dodge and Harry G Romig. 1929.  A method of sampling inspection.  _The Bell System Technical Journal_ 8, 4 (1929), 613–631.
  * Doob (1953) J.L. Doob. 1953.  _Stochastic Processes_.  Wiley.  https://books.google.de/books?id=KvJQAAAAMAAJ
  * Drysdale (2021) Erik Drysdale. 2021.  The sum of a normal and a truncated normal.  Blog Post.  https://www.erikdrysdale.com/NTS/
  * Gordon Lan and Demets (1983) K. K. Gordon Lan and David L. Demets. 1983.  Discrete sequential boundaries for clinical trials.  _Biometrika_ 70, 3 (12 1983), 659–663.  https://doi.org/10.1093/biomet/70.3.659 arXiv:https://academic.oup.com/biomet/article-pdf/70/3/659/686493/70-3-659.pdf
  * Ham et al. (2023) Dae Woong Ham, Iavor Bojinov, Michael Lindon, and Martin Tingley. 2023.  Design-Based Confidence Sequences: A General Approach to Risk Mitigation in Online Experimentation.  arXiv:2210.08639 [stat.ME]
  * Howard et al. (2021) Steven R. Howard, Aaditya Ramdas, Jon McAuliffe, and Jasjeet Sekhon. 2021.  Time-uniform, nonparametric, nonasymptotic confidence sequences.  _The Annals of Statistics_ 49, 2 (apr 2021).  https://doi.org/10.1214/20-aos1991
  * Jamieson and Jain (2018) Kevin Jamieson and Lalit Jain. 2018.  A Bandit Approach to Multiple Testing with False Discovery Control. In _Proceedings of the 32nd International Conference on Neural Information Processing Systems_ _(NIPS’18)_. Curran Associates Inc., Red Hook, NY, USA, 3664–3674.
  * Jamieson et al. (2013) Kevin Jamieson, Matthew Malloy, Robert Nowak, and Sébastien Bubeck. 2013.  lil’ UCB : An Optimal Exploration Algorithm for Multi-Armed Bandits.  arXiv:1312.7308 [stat.ML]
  * Johari et al. (2022) Ramesh Johari, Pete Koomen, Leonid Pekelis, and David Walsh. 2022.  Always valid inference: Continuous monitoring of a/b tests.  _Operations Research_ 70, 3 (2022), 1806–1821.
  * Kaufmann et al. (2016) Emilie Kaufmann, Olivier Cappé, and Aurélien Garivier. 2016.  On the complexity of best arm identification in multi-armed bandit models.  _Journal of Machine Learning Research_ 17 (2016), 1–42.
  * Kim (2006) Hea-Jung Kim. 2006.  On the Distribution and Its Properties of the Sum of a Normal and a Doubly Truncated Normal.  _The Korean Communications in Statistics_ (2006).
  * Kurennoy and Gurbanov (2024) Alexey Kurennoy and Tural Gurbanov. 2024.  YEAST: YEt Another Sequential Test.  Git Repository.  https://github.com/akurennoy/yeast.git This paper’s code companion.
  * Larsen et al. (2023) Nicholas Larsen, Jonathan Stallrich, Srijan Sengupta, Alex Deng, Ron Kohavi, and Nathaniel Stevens. 2023.  Statistical Challenges in Online Controlled Experiments: A Review of A/B Testing Methodology.  arXiv:2212.11366 [stat.AP]
  * Lindon et al. (2022a) Michael Lindon, Dae Woong Ham, Martin P. Tingley, and Iavor Bojinov. 2022a.  Anytime-Valid Linear Models and Regression Adjusted Causal Inference in Randomized Experiments.  https://api.semanticscholar.org/CorpusID:259309472
  * Lindon and Malek (2022) Michael Lindon and Alan Malek. 2022.  Anytime-Valid Inference For Multinomial Count Data. In _Advances in Neural Information Processing Systems_ , S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (Eds.), Vol. 35. Curran Associates, Inc., 2817–2831.  https://proceedings.neurips.cc/paper_files/paper/2022/file/12f3bd5d2b7d93eadc1bf508a0872dc2-Paper-Conference.pdf
  * Lindon et al. (2022b) Michael Lindon, Chris Sanden, and Vaché Shirikian. 2022b.  Rapid Regression Detection in Software Deployments through Sequential Testing. In _Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining_ (Washington DC, USA) _(KDD ’22)_. Association for Computing Machinery, New York, NY, USA, 3336–3346.  https://doi.org/10.1145/3534678.3539099
  * Loeve (1960) M. Loeve. 1960.  _Probability Theory. 2nd Edition_.  D. Van Nostrand Company Inc.
  * Miller (2015) Evan Miller. 2015.  Simple Sequential A/B Testing.  Blog Post.  https://www.evanmiller.org/sequential-ab-testing.html
  * O’Brien and Fleming (1979) Peter C. O’Brien and Thomas R. Fleming. 1979.  A Multiple Testing Procedure for Clinical Trials.  _Biometrics_ 35, 3 (1979), 549–556.  http://www.jstor.org/stable/2530245
  * Pocock (1977) Stuart J. Pocock. 1977.  Group Sequential Methods in the Design and Analysis of Clinical Trials.  _Biometrika_ 64, 2 (1977), 191–199.  http://www.jstor.org/stable/2335684
  * Robbins (1970) Herbert Robbins. 1970.  Statistical Methods Related to the Law of the Iterated Logarithm.  _The Annals of Mathematical Statistics_ 41, 5 (1970), 1397 – 1409.  https://doi.org/10.1214/aoms/1177696786
  * Schultzberg and Ankargren (2023a) Mårten Schultzberg and Sebastian Ankargren. 2023a.  Choosing a Sequential Testing Framework — Comparisons and Discussions.  Blog Post.  https://engineering.atspotify.com/2023/03/choosing-sequential-testing-framework-comparisons-and-discussions/
  * Schultzberg and Ankargren (2023b) Mårten Schultzberg and Sebastian Ankargren. 2023b.  Simulation files for Schultzberg & Ankargren blogpost 2023.  GitHub repository.  https://github.com/MSchultzberg/SeqTestBlog
  * Slud and Wei (1982) Eric V. Slud and L. J. Wei. 1982.  Two-Sample Repeated Significance Tests Based on the Modified Wilcoxon Statistic.  _J. Amer. Statist. Assoc._ 77 (1982), 862–868.  https://api.semanticscholar.org/CorpusID:121593359
  * STATSIG (2023) STATSIG. 2023.  Sequential Testing.  Documentation.  https://docs.statsig.com/experiments-plus/sequential-testing
  * van der Vaart (2000) A.W. van der Vaart. 2000.  _Asymptotic Statistics_.  Cambridge University Press.  https://books.google.ie/books?id=SYlmEAAAQBAJ
  * Wald (1945) A. Wald. 1945.  Sequential Tests of Statistical Hypotheses.  _The Annals of Mathematical Statistics_ 16, 2 (1945), 117 – 186.  https://doi.org/10.1214/aoms/1177731118
  * Wald (1947) A. Wald. 1947.  _Sequential analysis_.  Wiley.  https://books.google.de/books?id=0nREAAAAIAAJ
  * Wassmer and Brannath (2016) G. Wassmer and W. Brannath. 2016.  _Group Sequential and Confirmatory Adaptive Designs in Clinical Trials_.  Springer International Publishing.  https://books.google.ie/books?id=hT2lDAAAQBAJ
  * White (1984) H. White. 1984.  _Asymptotic Theory for Econometricians_.  Elsevier Science.  https://books.google.ie/books?id=hSu7AAAAIAAJ
  * Yang et al. (2017) Fanny Yang, Aaditya Ramdas, Kevin Jamieson, and Martin Wainwright. 2017.  A Framework for Multi-A(Rmed)/B(Andit) Testing with Online FDR Control. In _Proceedings of the 31st International Conference on Neural Information Processing Systems_ _(NIPS’17)_. Curran Associates Inc., Red Hook, NY, USA, 5959–5968.
  * Zeileis (2004) Achim Zeileis. 2004.  Econometric Computing with HC and HAC Covariance Matrix Estimators.  _Journal of Statistical Software_ 11, 10 (2004), 1–17.  https://doi.org/10.18637/jss.v011.i10
  * Zeileis et al. (2020) Achim Zeileis, Susanne Köll, and Nathaniel Graham. 2020.  Various Versatile Variances: An Object-Oriented Implementation of Clustered Covariances in R.  _Journal of Statistical Software_ 95, 1 (2020), 1–36.  https://doi.org/10.18637/jss.v095.i01

## Appendix

##  Appendix A Proofs and Auxiliary Statements

###### Lemma 0.

Let $Y_{i}\colon\Omega\mapsto\mathrm{I}\\!\mathrm{R}$ and
$W_{i}\colon\Omega\mapsto\\{0,\,1\\}$, $i=1,\,\ldots,\,N$, be random
variables, and let $S_{n}$, $n\geq 1$, be the running sum defined by (1)–(2).
Suppose that Assumption 1 and the null hypothesis (3)–(4) hold.

Then for any $N\geq 1$ and any $k=1,\,\ldots,\,N$ the distribution of
$S_{N}-S_{k}$ conditional on $S_{1},\,\ldots,\,S_{k}$ is symmetric around 0.

###### Proof.

Fix any $N\geq 2$ and any $k=1,\,\ldots,\,N-1$ (for $N=1$ or $k=N$ the
statement is trivial). Let $\mathcal{T}=\\{-1,\,+1\\}^{N-k}$ be the set of all
possible combinations of $-1$ and $+1$ of length $N-k$. Furthermore, for any
$\tau\in\mathcal{T}$, let $\bar{\tau}\in\mathcal{T}$ be the combination
obtained by negating every element of $\tau$, i. e. $\bar{\tau}_{j}=-\tau_{j}$
for all $j=1,\,\ldots,\,N-k$. Next, define

$A_{\tau}=\\{1-2W_{i}=\tau_{i-k}\
\forall\,i=k+1,\,\ldots,\,N\\},\quad\tau\in\mathcal{T}.$

Note that

(10) $A_{\tau}\cap A_{\tau^{\prime}}=\emptyset\
\forall\,\tau,\,\tau^{\prime}\in\mathcal{T}\text{ and
}\cup_{\tau\in\mathcal{T}}A_{\tau}=\cup_{\tau\in\mathcal{T}}A_{\bar{\tau}}=\Omega.$

Also, Assumption 1 together with the null hypothesis imply that

(11) $\Pr(A_{\tau})=\Pr(A_{\bar{\tau}})\quad\forall\,\tau\in\mathcal{T}.$

Finally, fix an arbitrary Borel set $Q\subset\mathrm{I}\\!\mathrm{R}$ and set

$B=\\{S_{N}-S_{k}\in Q\\},\quad\bar{B}=\\{S_{k}-S_{N}\in Q\\}.$

Denoting the conditional probability $\Pr(\cdot\mid S_{1},\,\ldots,\,S_{k})$
by $\Pr_{k}$, we will now derive the key relationship for the proof of the
lemma. Specifically, (3) and (11) imply that for any $\tau\in\mathcal{T}$ we
have

(12) $\Pr\nolimits_{k}(\bar{B}\cap
A_{\tau})=\Pr\nolimits_{k}\left(\left\\{-\sum_{i=k+1}^{N}\tau_{i-k}Y_{i}\in
Q\right\\}\cap A_{\tau}\right)\stackrel{{\scriptstyle\eqref{eq:H0a}}}{{=}}\\\
\stackrel{{\scriptstyle\eqref{eq:H0a}}}{{=}}\Pr\nolimits_{k}\left\\{-\sum_{i=k+1}^{N}\tau_{i-k}Y_{i}\in
Q\right\\}\Pr\nolimits_{k}(A_{\tau})\stackrel{{\scriptstyle\eqref{eq:PAtau}}}{{=}}\\\
\stackrel{{\scriptstyle\eqref{eq:PAtau}}}{{=}}\Pr\nolimits_{k}\left\\{-\sum_{i=k+1}^{N}\tau_{i-k}Y_{i}\in
Q\right\\}\Pr\nolimits_{k}(A_{\bar{\tau}})=\\\
=\Pr\nolimits_{k}\left\\{\sum_{i=k+1}^{N}\bar{\tau}_{i-k}Y_{i}\in
Q\right\\}\Pr\nolimits_{k}(A_{\bar{\tau}})\stackrel{{\scriptstyle\eqref{eq:H0a}}}{{=}}\\\
\stackrel{{\scriptstyle\eqref{eq:H0a}}}{{=}}\Pr\nolimits_{k}\left(\left\\{\sum_{i=k+1}^{N}\bar{\tau}_{i-k}Y_{i}\in
Q\right\\}\cap A_{\bar{\tau}}\right)=\Pr\nolimits_{k}(B\cap A_{\bar{\tau}})$

From (12) and (10), it follows that

(13)
$\Pr\nolimits_{k}(\bar{B})\stackrel{{\scriptstyle\eqref{eq:=Omega}}}{{=}}\sum_{\tau\in\mathcal{T}}\Pr\nolimits_{k}(\bar{B}\cap
A_{\tau})\stackrel{{\scriptstyle\eqref{eq:swap}}}{{=}}\\\
\stackrel{{\scriptstyle\eqref{eq:swap}}}{{=}}\sum_{\tau\in\mathcal{T}}\Pr\nolimits_{k}(B\cap
A_{\bar{\tau}})\stackrel{{\scriptstyle\eqref{eq:=Omega}}}{{=}}\Pr\nolimits_{k}(B)$

Since the fixed Borel set $Q$ was arbitrary, (13) proves that conditionally on
$S_{1},\,\ldots,\,S_{k}$, the distribution of $S_{N}-S_{k}$ and $S_{k}-S_{N}$
is the same or, equivalently, the distribution of $S_{N}-S_{k}$ is symmetric
about zero. ∎

###### Lemma 0.

Suppose that the assumptions of Lemma 1 hold.

Then for any $N\geq 1$ and any $b>0$, we have

(14) $\Pr\left\\{\max_{n=1}^{N}S_{n}\geq b\right\\}\leq 2\Pr\\{S_{N}\geq
b\\},$

and

(15) $\Pr\left\\{\max_{n=1}^{N}|S_{n}|\geq b\right\\}\leq 2\Pr\\{|S_{N}|\geq
b\\}.$

###### Proof.

From Lemma 1, it follows that for any $k=1,\,\ldots,\,N$, the distribution of
$S_{N}-S_{k}$ conditional on $S_{1},\,\ldots,\,S_{k}$ is symmetric about 0 and
therefore the conditional median $\mu(S_{N}-S_{k}\mid S_{1},\,\ldots,\,S_{k})$
is zero. Then (14) and (15) easily follow from the generalised Levy’s
inequalities (Loeve, 1960, Paragraph 29.1, A). ∎

###### Proof of Theorem 1..

Fix an arbitrary $\alpha\in[0,\,1]$ and $\varepsilon>0$.

By the continuity of $\Phi$, there exists a sufficiently small $\delta>0$ such
that

(16)
$\left|\Phi\left(z_{1-\alpha/2}\right)-\Phi\left(z_{1-\alpha/2}-\delta\right)\right|\leq\frac{\varepsilon}{4}.$

Furthermore, the null hypothesis implies that

$\mathop{\textrm{E}}[X_{i}]=\mathop{\textrm{E}}\left[(1-2W_{i})Y_{i}\right]\stackrel{{\scriptstyle\eqref{eq:H0a}}}{{=}}(1-2\Pr\\{W_{i}=1\\})\mathop{\textrm{E}}[Y_{i}]\stackrel{{\scriptstyle\eqref{eq:H0b}}}{{=}}0$

for all $i\geq 1$. Therefore, $E[S_{N}]=0$ for all $N\geq 1$ and by the
Central Limit Theorem we have that there exists $N_{\alpha,\,\varepsilon}\geq
1$ such that

(17) $\left|\Pr\left\\{S_{N}/\sqrt{var(S_{N})}\leq
z_{1-\alpha/2}-\delta\right\\}-\Phi\left(z_{1-\alpha/2}-\delta\right)\right|\leq\frac{\varepsilon}{4}$

for all $N\geq N_{\alpha,\,\varepsilon}$. Together with (14), relations
(16)–(17) give

$\Pr\left\\{\max_{n=1}^{N}S_{n}>z_{1-\alpha/2}\sqrt{NV_{N}}\right\\}=\\\
=Pr\left\\{\max_{n=1}^{N}S_{n}>z_{1-\alpha/2}\sqrt{var(S_{N})}\right\\}\leq\\\
\leq\Pr\left\\{\max_{n=1}^{N}S_{n}\geq
z_{1-\alpha/2}\sqrt{var(S_{N})}\right\\}\stackrel{{\scriptstyle\eqref{eq:levy1}}}{{\leq}}\\\
\stackrel{{\scriptstyle\eqref{eq:levy1}}}{{\leq}}2\Pr\left\\{S_{N}\geq
z_{1-\alpha/2}\sqrt{var(S_{N})}\right\\}=\\\
=2\Pr\left\\{S_{N}/\sqrt{var(S_{N})}\geq z_{1-\alpha/2}\right\\}\leq\\\ \leq
2\Pr\left\\{S_{N}/\sqrt{var(S_{N})}>z_{1-\alpha/2}-\delta\right\\}=\\\
=2\left(1-\Pr\left\\{S_{N}/\sqrt{var(S_{N})}\leq
z_{1-\alpha/2}-\delta\right\\}\right)=\\\
=2\left(1-\Phi\left(z_{1-\alpha/2}\right)\right)+\\\
+2\left(\Phi\left(z_{1-\alpha/2}\right)-\Phi\left(z_{1-\alpha/2}-\delta\right)\right)+\\\
+2\left(\Phi\left(z_{1-\alpha/2}-\delta\right)-\Pr\left\\{S_{N}/\sqrt{var(S_{N})}\leq
z_{1-\alpha/2}-\delta\right\\}\right)\stackrel{{\scriptstyle\eqref{eq:continuity},\,\eqref{eq:clt}}}{{\leq}}\\\
\stackrel{{\scriptstyle\eqref{eq:continuity},\,\eqref{eq:clt}}}{{\leq}}\alpha+\frac{\varepsilon}{2}+\frac{\varepsilon}{2}=\alpha+\varepsilon$

for all $N\geq N_{\alpha,\,\varepsilon}$. This completes the proof. ∎

###### Proof of Theorem 2..

Denote $m_{N}=\sum_{i=1}^{N}\mu_{i}$ and let $\xi_{N}$ stand for the centered
and standardised sum $(S_{N}-N\mu)/var(S_{N})$. By the Central Limit Theorem
$\xi_{N}\stackrel{{\scriptstyle d}}{{\to}}\mathcal{N}(0,\,1)$ and therefore
$\xi_{N}$ is stochastically bounded (see, for example, (van der Vaart, 2000,
Theorem 2.4)).

Now, fix an arbitrary $\varepsilon>0$. From the stochastical boundedness of
$\xi_{N}$ and $V_{N}$, it follows that there exist $M_{1}$ and $M_{2}$ such
that for all sufficiently large $N$ we have

(18) $\Pr\left\\{|\xi_{N}|\leq M_{1}\right\\}\geq 1-\frac{\varepsilon}{2}$

and

(19) $\Pr\left\\{|V_{N}|\leq M_{2}\right\\}\geq 1-\frac{\varepsilon}{2}.$

From (18)–(19), it follows that

$\Pr\left\\{\xi_{N}+m_{N}\sqrt{N}/\sqrt{V_{N}}\geq-
M_{1}+m_{N}\sqrt{N}/\sqrt{M_{2}}\right\\}\geq 1-\varepsilon$

for all large enough $N$. But then for any sufficiently large $N$, we have
that

(20)
$\Pr\left\\{\xi_{N}+m_{N}\sqrt{N}/\sqrt{V_{N}}>z_{1-\alpha/2}\right\\}\geq
1-\varepsilon.$

because $\left(-M_{1}+m_{N}\sqrt{N}/\sqrt{M_{2}}\right)\to\infty$ as
$N\to\infty$. Then we can derive that

$\Pr\left\\{\max_{n=1}^{N}S_{n}>z_{1-\alpha/2}\sqrt{NV_{N}}\right\\}\geq\Pr\left\\{S_{N}>z_{1-\alpha/2}\sqrt{NV_{N}}\right\\}=\\\
=\Pr\left\\{\xi_{N}+m_{N}\sqrt{N}/\sqrt{V_{N}}>z_{1-\alpha/2}\right\\}\stackrel{{\scriptstyle\eqref{eq:ge1minuseps}}}{{\geq}}1-\varepsilon$

for all sufficiently large $N$. Since $\varepsilon$ was arbitrary, this
completes the proof. ∎

##  Appendix B Additional Evaluations

In this appendix, we report additional evaluation results for the case where
the evaluated methods (see the list in Section 5) were employed in the
discrete mode (i. e. with only a fixed number of interim checks). The
evaluation were performed against the same replications an in our main
experiment in Section 5. We again follow the protocol from (Schultzberg and
Ankargren, 2023a) and perform 14, 28, 42, and 56 significance checks (spaced
equally across the timeline). Since in these evaluations, we operate in a
discrete setting, we were able to include the discrete baselines from
(Schultzberg and Ankargren, 2023a) in the comparison. Namely, we additionally
compare against the following method.

GST The group sequential test with alpha spending as proposed in (Gordon Lan
and Demets, 1983). The test performs a significance check after the arrival of
each batch of observations. To schedule a prespecified number of checks it
therefore needs an estimate of the total number of observations that would be
collected during the experiment time frame. In the evaluations, we consider
three different scenarios: when the sample size is estimated precisely (right
number of checks), when the sample size is underestimated (leading to more
checks that planned), and when the sample size is overstimated (leading to
making fewer checks than planned). The respective entries in the evaluation
table are referred to as _GST_ , _GSToversampled_ , and _GSTundersampled_ ,
respectively. The actual sample size was 500 and the respective sample size
estimates for the three scenarios were 500, 250, and 750. In the case of
oversampling (i. e. the sample size is underestimated), we apply the
correction to the bounds proposed in (Wassmer and Brannath, 2016, pp. 78–79).
We consider quadratic and cubic alpha-spending, the latter having the “phi3”
prefix in the name.

We report the share of replications where the test detects an effect (i. e.
the significance is detected in at least one of the interim checks). Table 5
(“False Positive Rate”) reports this share for the case where no actual effect
was present. All of the compared methods except CAA and oversampled versions
of GST keep the false detection rate below the nominal level of 5%. Table 6
(“Power”) contains the share of replications with a detection for the case
where the treatment effect was set to 0.2 standard deviations. The methods
with an inflated false detection rate were excluded from the power
comparisons. The GST method with cubic alpha-spending demonstrated the highest
power, closely followed by SST (the method proposed in Section 3), and GST
with quadratic alpha-spending. We find it remarkable that our proposed method,
despite supporting continouous monitoring, performed on par with the GST
method in the discrete monitoring setting.

Table 5. False Positive Rate | type | 14 | 28 | 42 | 56
---|---|---|---|---|---
1 | YEAST | 0.04 | 0.04 | 0.04 | 0.04
2 | pYEAST7 | 0.02 | 0.03 | 0.03 | 0.03
3 | pYEAST14 | 0.02 | 0.02 | 0.02 | 0.02
4 | GST | 0.05 | 0.05 | 0.05 | 0.05
5 | GSTphi3 | 0.05 | 0.05 | 0.05 | 0.05
6 | GSToversampled | 0.07 | 0.07 | 0.07 | 0.08
7 | GSToversampledphi3 | 0.09 | 0.10 | 0.10 | 0.07
8 | GSTundersampled | 0.03 | 0.02 | 0.03 | 0.03
9 | GSTundersampledphi3 | 0.01 | 0.01 | 0.01 | 0.01
10 | mSPRTphi100 | 0.01 | 0.01 | 0.01 | 0.01
11 | mSPRTphi11 | 0.01 | 0.02 | 0.02 | 0.02
12 | mSPRTphi25 | 0.01 | 0.02 | 0.02 | 0.02
13 | GAVI250 | 0.01 | 0.02 | 0.02 | 0.02
14 | GAVI500 | 0.01 | 0.01 | 0.01 | 0.01
15 | GAVI750 | 0.01 | 0.01 | 0.01 | 0.01
16 | CAA | 0.06 | 0.06 | 0.07 | 0.07
17 | Bonferroni | 0.01 | 0.01 | 0.01 | 0.01
Table 6. Power (under a treatment effect of 0.2 standard deviations) | type | 14 | 28 | 42 | 56 | stream
---|---|---|---|---|---|---
1 | YEAST | 0.90 | 0.91 | 0.91 | 0.91 | 0.92
2 | pYEAST7 | 0.79 | 0.80 | 0.81 | 0.81 | 0.82
3 | pYEAST14 | 0.75 | 0.76 | 0.76 | 0.77 | 0.78
4 | GST | 0.90 | 0.90 | 0.90 | 0.89 | -
5 | GSTphi3 | 0.93 | 0.92 | 0.93 | 0.93 | -
6 | GSTundersampled | 0.83 | 0.82 | 0.82 | 0.82 | -
7 | GSTundersampledphi3 | 0.72 | 0.71 | 0.71 | 0.71 | -
8 | mSPRTphi100 | 0.71 | 0.73 | 0.73 | 0.74 | 0.76
9 | mSPRTphi11 | 0.68 | 0.70 | 0.70 | 0.71 | 0.73
10 | mSPRTphi25 | 0.71 | 0.72 | 0.73 | 0.74 | 0.76
11 | GAVI250 | 0.72 | 0.73 | 0.74 | 0.75 | 0.76
12 | GAVI500 | 0.72 | 0.73 | 0.74 | 0.74 | 0.76
13 | GAVI750 | 0.71 | 0.72 | 0.73 | 0.73 | 0.75
14 | Bonferroni | 0.55 | 0.53 | 0.51 | 0.50 | 0.40