# Online multiple hypothesis testing for reproducible research

David S. Robertson  MRC Biostatistics Unit, University of Cambridge, UK  James
M. S. Wason  Population Health Sciences Institute, Newcastle University,
Newcastle, UK  Aaditya Ramdas  Departments of Statistics and Machine Learning,
Carnegie Mellon University, Pittsburgh, Pennsylvania, USA

###### Abstract

Modern data analysis frequently involves large-scale hypothesis testing, which
naturally gives rise to the problem of maintaining control of a suitable type
I error rate, such as the false discovery rate (FDR). In many biomedical and
technological applications, an additional complexity is that hypotheses are
tested in an online manner, one-by-one over time. However, traditional
procedures that control the FDR, such as the Benjamini-Hochberg procedure,
assume that all $p$-values are available to be tested at a single time point.
To address these challenges, a new field of methodology has developed over the
past 15 years showing how to control error rates for online multiple
hypothesis testing. In this framework, hypotheses arrive in a stream, and at
each time point the analyst decides whether to reject the current hypothesis
based both on the evidence against it, and on the previous rejection
decisions. In this paper, we present a comprehensive exposition of the
literature on online error rate control, with a review of key theory as well
as a focus on applied examples. We also provide simulation results comparing
different online testing algorithms and an up-to-date overview of the many
methodological extensions that have been proposed.

Keywords: A/B testing, data repositories, online multiple hypothesis testing,
platform trials, type I error rate

##  1 Introduction

Large-scale hypothesis testing is now ubiquitous in a variety of biomedical
and technological applications. For example, many major technology companies
perform tens of thousands of randomised controlled experiments (known as A/B
tests) each year to make data-driven decisions about how to improve products
[Kohavi et al., 2020]. Meanwhile, in genomics it is now routine to test
hundreds of thousands of genetic variants for an association with particular
phenotypic trait(s). Even in the setting of randomised controlled trials
(RCTs) in medicine, there is a growing push towards the use of “overarching”
trial frameworks to allow the efficient testing of multiple experimental drugs
for multiple patient subpopulations.

Performing a large number of hypothesis tests naturally gives rise to the
problem of multiple comparisons [Tukey, 1953]: given a collection of multiple
hypotheses to be tested, the goal is to distinguish which hypotheses are null
and non-null, while controlling a suitable error rate (see Section 1.2). This
error rate is generally formed around the probability of incorrectly
classifying a null hypothesis as non-null. Typically, a $p$-value is
calculated for each hypothesis and is then used to decide whether to reject
the null hypothesis. Multiple hypothesis testing is one of the core problems
in statistical inference, and has led to a wide range of procedures that can
be used to correct for multiplicity and ensure that a suitable error rate is
controlled. In contrast, uncorrected hypothesis testing contributes to serious
concerns over reproducibility, publication bias and ‘p-hacking’ in scientific
research [Ioannidis, 2005, Head et al., 2015].

Traditionally, multiple hypothesis testing is offline in nature, in the sense
that a procedure for testing $N$ hypotheses will receive all of the
corresponding $p$-values $(p_{1},\ldots,p_{N})$ at once. Step-up and step-down
multiple testing procedures (for example) require knowledge about all
$p$-values in advance. However, this paradigm is often incompatible with
modern data-driven decision-making processes, as demonstrated by our
motivating examples in Section 1.1. Once the data is available to make a
decision about a particular hypothesis, it can be desirable to take a
corresponding action (e.g. to update a tech product) relatively quickly, and
not to wait for the results of ongoing or future hypothesis tests. Linked with
this, in many application areas it may not even be possible to know in advance
how many tests in total will be performed. Moreover, the repeated application
of traditional offline multiple testing procedures as the family of hypotheses
grows can lead to repeatedly changing past decisions, which is clearly
undesirable from the practical perspective of wanting to act in a timely
manner on test results.

What is needed therefore are procedures for online multiple hypothesis
testing, which better take into account the nature of modern data analysis.
This is defined as follows: A stream of hypotheses arrives online. At each
step, the analyst must decide whether to reject the current null hypothesis
without having access to the number of hypotheses (potentially infinite) or
any future data, but solely based on the previous decisions and evidence
against the current hypothesis.

Online hypothesis testing has a sequential nature, in the sense that
individual hypotheses (or batches of hypotheses) are tested one after the
other over time. However, this is distinct from the more traditional concept
of sequential testing, which refers to the testing of a single hypothesis in a
sequential manner. In sequential testing, the sample size for the experiment
is not fixed in advance, and the accumulating data is evaluated as they are
collected to allow the experiment to be stopped adaptively, such as when
statistical significance is achieved. The framework of online multiple testing
can be expanded to be “doubly sequential”, where the inner sequential process
is a single sequential test, and the outer sequential process refers to the
multiple experiments that are performed to test different hypotheses.

We also wish to draw a distinction between online hypothesis testing and
multi-armed bandit (MAB) testing. While both frameworks allow the comparison
of multiple experimental arms over time, an MAB can be considered as a single
experiment in which resources are iteratively allocated to the different arms
in order to adaptively trade off certain costs and benefits, and this
allocation depends on the previously observed outcomes on each arm. Again, the
two testing frameworks can be combined within a doubly sequential framework
where there is a sequence of MAB problems over time, see Yang et al. [2017].

Figure 1 gives a diagrammatic representation of online multiple testing, where
different hypotheses (corresponding to experiments) are tested over time
(corresponding to the collection of data samples). As discussed above, each
experiment could itself be a sequential experiment or take the form of an MAB.

Figure 1: An abstract online multiple testing framework. As time passes (left
to right), new experiments testing different hypotheses are started and
stopped, in a possibly indefinite manner. Each horizontal line represents a
new experiment/hypothesis, and the length represents the number of samples
collected. Decisions about each hypothesis must be made as soon as the
corresponding experiment ends.

Since the framework was first proposed by Foster and Stine [2008], a variety
of procedures that control error rates for online hypothesis testing have been
developed [Aharoni and Rosset, 2014, Javanmard and Montanari, 2015, Ramdas et
al., 2018]. Our aim in this paper is to provide an expository overview of this
literature on online error rate control, with a review of the underlying
theory, key methods and applied examples.

In the rest of this section, we give motivating examples for online multiple
testing and present formal definitions of error rate control. Section 2
describes the key procedures for online error rate control in detail. Section
3 presents a simulation study of the procedures, while Section 4 presents two
case studies of applying online error rate control. We describe further
methodological extensions as well as future directions in Section 5, and
conclude with a discussion in Section 6.

###  1.1 Motivating examples

A/B testing in tech companies
The development of web applications and services in the tech industry
increasingly relies on the use of randomised controlled experiments known as
A/B tests. There are a number of widely used platforms now available that
streamline and handle the implementation of A/B tests. A typical application
is in the development of different versions of webpages. As described in
Berman and Van den Bulte [2021], in this context there are two webpage
variations (A and B). When an online user visits, the platform randomly
assigns the visitor to one of the variations for the duration of the
experiment. The platform records the actions that the visitor takes, where the
monitored actions reflect the experimenter’s goal(s), such as increasing
visitor engagement (defined appropriately) or increasing revenue. One of the
variations is designated as the baseline, and the performance of the other
variation is compared to the baseline using suitable test statistics. If run
correctly using anytime-valid p-values [Johari et al., 2021] and/or confidence
sequences [Howard et al., 2021], the data can be continuously monitored by the
experimenter, allowing a decision to be made at any stopping time.

Many tech companies run tens of thousands of A/B tests per year, as part of a
continuous process of designing, delivering, monitoring and improving webpages
and other web services. However, there is reason to reduce the number of false
alarms which result in making changes to web products that do no better (or
even perform worse) than the current iteration, corresponding to an incorrect
rejection of the null, particularly when such changes are potentially costly
or disruptive to users. Hence, the framework of online error rate control
provides a framework to do so while still allowing a large number of A/B tests
to be performed in a flexible manner.

Data repositories
Public databases and shared data resources are becoming increasingly pervasive
and important in modern biomedical research, particularly in the fields of
genetics, molecular biology and routinely collected healthcare records. Some
well-known examples include the 1000 Genomes Project [1000 Genomes Project
Consortium, 2015] and the Wellcome Trust Case Control Consortium [Wellcome
Trust Case Control Consortium et al., 2007]. Another example is the
International Mouse Phenotyping Consortium database [Koscielny et al., 2013,
Dickinson et al., 2016], which we describe as one of our case studies in
Section 4. Meanwhile, the increase in routinely collected healthcare data has
allowed evaluation of different healthcare technologies currently used in
practice through emulation of target trials [Dickerman et al., 2019].

Multiple testing naturally occurs in this setting in two ways. Firstly, such
databases can be accessed by multiple independent researchers at different
times. When a researcher or research group comes up with a new hypothesis,
they can fetch the relevant data from a database and perform a statistical
test. Secondly, in many databases the family of hypotheses to be tested grows
over time as new data is continually added. In both of these scenarios, the
number of hypotheses being tested will be unknown and potentially very large.

In order to control the number or proportion of false discoveries in this
context, new procedures are required that allow a researcher to decide whether
to reject a current hypothesis with minimal information about previous
hypotheses, and without prior knowledge of even the number of hypotheses that
are going to be tested in the future. This is precisely the online multiple
testing framework described earlier.

Platform trials
A platform trial has a single master protocol that evaluates multiple
treatments across one or more patient types, and allows a potentially large
number of treatments to be added during the course of the trial [Saville and
Berry, 2016]. Treatments are dropped from the trial after they have been
formally tested for effectiveness. Such a trial could (in theory) ‘perpetual’
in that new treatments can continue to enter into the trial and be tested.
Figure 4 in Section 4.2 gives a diagram of an example platform trial showing
what this looks like in practice.

In a platform trial, treatments are introduced at different time points by
design. However, the trial investigators will wish to make a decision on
whether a treatment is beneficial as soon as the data are ready, without
waiting for results from the other treatment arms. Hence, the treatment
effects are tested sequentially in an online manner, where the number of
treatments to be tested in the future may be unknown. More formally, a
platform trial generates a sequence of null hypotheses
$(H_{1},H_{2},H_{3},\ldots)$ which are tested sequentially. Hypothesis $H_{i}$
tests the value of some parameter $\theta_{i}$, such as an estimate of the
treatment difference compared to a control arm. Note that the control arm
could also change during the trial, for example due to one of the previously
evaluated experimental arms being confirmed as effective and becoming the new
control.

The $p$-values generated from the platform trial described above will not be
independent in general. Dependencies will primarily arise due to the shared
control data that is re-used to test multiple hypotheses. A current example of
a long-running platform trial is the STAMPEDE trial [James et al., 2008] for
patients with locally advanced or metastatic prostate cancer, which we return
to as a case study in Section 4.2.

###  1.2 Error rates

We now formally define some error rates of interest. The basic problem setup
is as follows. At each time step $t=1,2,\ldots$ the experimenter observes a
$p$-value $P_{t}$ corresponding to a null hypothesis $H_{t}$, and must make a
decision whether to reject $H_{t}$ before the next time step. We assume that
all $p$-values are valid, i.e. if the null hypothesis $H_{t}$ is true, then
$\text{Pr}\\{P_{t}\leq x\\}\leq x$ for all $x\in[0,1]$. At time $t=0$, the
experimenter fixes the level $\alpha$ at which a suitable error rate is meant
to be controlled at all times.

A general testing procedure provides a sequence of test levels $\alpha_{t}$
with decision rule

$R_{t}=\begin{cases}1\qquad\text{if }P_{t}\leq\alpha_{t}\qquad\text{(reject
}H_{t})\\\ 0\qquad\text{otherwise}\qquad\text{(accept }H_{t})\end{cases}~{}.$
(1)

At any time $T$, let $R(T)=\sum_{t=1}^{T}R_{t}$ denote the number of
rejections (also known as discoveries) made so far and $V(T)$ denote the total
number of falsely rejected hypotheses (also known as false discoveries).

The false discovery proportion at time $T$ is defined as

$\text{FDP}(T):=\frac{V(T)}{R(T)\vee 1},$

where $a\vee b=\max(a,b)$. The false discovery rate is then the expectation of
the FDP:

$\text{FDR}(T):=\mathbb{E}\\!\left[\frac{V(T)}{R(T)\vee 1}\right].$

A commonly studied variant is the marginal FDR:

$\text{mFDR}(T):=\frac{\mathbb{E}[V(T)]}{\mathbb{E}[R(T)\vee 1]}.$

Another related error rate is the false discovery exceedance (FDX), which is
the probability the FDP exceeds a predefined threshold $\epsilon$:

$\text{FDX}_{\epsilon}(T)=\text{Pr}\left[\text{sup}_{t}\,\text{FDP}(t)\geq\epsilon\right].$

An alternative error rate to those based on the FDP is the familywise error
rate (FWER), which is more commonly considered in clinical trial contexts. The
FWER is the probability of falsely rejecting any null hypothesis:

$\text{FWER}(T)=\text{Pr}\left[V(T)\geq 1\right].$

The FWER and hence the FDR can be controlled at level $\alpha$ in a simple
manner by using a Bonferonni-type correction, also known as alpha-spending.
More precisely, we can choose significance levels $\alpha_{t}$ for $H_{t}$,
such that $\sum_{t=1}^{\infty}\alpha_{t}=\alpha$. However, this method suffers
from a low statistical power, with the probability of the null hypothesis
$H_{t}$ being rejected tending to zero as $t$ increases. This motivates the
development of more sophisticated algorithms for online error rate control.

##  2 Online error rate control methodology

###  2.1 Generalised alpha investing (GAI)

The first proposals for online error rate control were based on “alpha
investing” by Foster and Stine [2008] and its generalization (GAI) [Aharoni
and Rosset, 2014]. These have an intuitive interpretation, described next. Any
GAI rule begins with an error budget, or alpha-wealth, which is allocated to
the different hypothesis tests over time. That is, there is a price (or
penalty) to be paid each time a hypothesis is tested. However, when a new
discovery is made (i.e. a hypothesis is rejected) some of the alpha-wealth is
earned back, which can be viewed as a return on the alpha-investment. Hence as
long as discoveries continue to be made, hypotheses can be tested indefinitely
without the test levels tending towards zero.

Formally, a GAI rule produces a series of test levels
$(\alpha_{1},\alpha_{2},\alpha_{3},\ldots)$ based on which it uses (1) to
produce the corresponding decisions $(R_{1},R_{2},R_{3},\ldots)$. Of course,
$\alpha_{t}$ must be based only on $R_{1},\dots,R_{t-1}$. At each time point
$t$, the rule updates the alpha-wealth $W(t)$ as follows:

  * •

If hypothesis $H_{t}$ is not rejected ($R_{t}=0$), then the alpha-wealth is
decreased by $\phi_{t}$.

  * •

If hypothesis $H_{t}$ is rejected ($R_{t}=1$), then the alpha-wealth is
increased by $\psi_{t}-\phi_{t}$.

In other words, the penalty $\phi_{t}$ is the amount paid for testing a new
hypothesis, and the payout (or reward) $\psi_{t}$ is the amount earned if a
discovery is made at that time. Hence the initial wealth is $W(0)=w_{0}$ and
it is updated via:

$\displaystyle W(t)$ $\displaystyle=W(t-1)-\phi_{t}+R_{t}\psi_{t}.$

The parameter $w_{0}$ and sequences $\alpha_{t},\phi_{t},\psi_{t}$ are all
user-defined. The total wealth $W(t)$ must always be non-negative, and hence
$\phi_{t}\leq W(t-1)$. Additionally, there are restrictions on
$\alpha_{t},\phi_{t},\psi_{t}$ in order to control the FDR (or its variants),
namely that when a rejection is made, the payout $\psi_{t}$ is capped. As is
predominately the case in offline multiple testing, we often use monotone
decision rules for $\alpha_{t}$ considered as a function of
$(R_{1},\ldots,R_{t})$: if $\tilde{R}_{i}\geq R_{i}$ for all $i\leq t-1$, then
we have
$\alpha_{t}(\tilde{R}_{1},\ldots,\tilde{R}_{t-1})\geq\alpha_{t}(R_{1},\ldots,R_{t})$.

Ramdas et al. [2017] defined a class of improved GAI algorithms, called GAI++,
as follows. We set $w_{0}$ so that $0\leq w_{0}\leq\alpha$ and choose the
reward $\psi_{t}$ to satisfy
$\psi_{t}\leq\min\\{\phi_{t}+b_{t},\frac{\phi_{t}}{\alpha_{t}}+b_{t}-1\\}$,
where

$b_{t}=\begin{cases}\alpha-W_{0}\qquad\text{when }R(t-1)=0\\\
\alpha\qquad\qquad\;\;\;\text{otherwise}\end{cases}~{}.$

######  Theorem 2.1.

If the null $p$-values are independent of all other $p$-values, any monotone
GAI++ rule satisfies the bound
$\mathbb{E}\left[\frac{V(t)+W(t)}{R(t)}\right]\leq\alpha$ for all $t\geq 1$.
Since $W(t)\geq 0$, the FDR is controlled at level $\alpha$ under
independence.

###  2.2 Algorithms for online FDR control: LORD, SAFFRON and ADDIS

Although an “algorithmic perspective” led to the GAI procedures initially used
in the online testing literature, Ramdas et al. [2017] posited a “statistical
perspective” to construct procedures, which is to keep an estimate of the FDP
less than $\alpha$. First, the oracle FDP is defined as

$\text{FDP}^{*}(t)=\frac{\sum_{j\leq
t,j\in\mathcal{H}_{0}}\alpha_{j}}{R(t)\vee 1},$

where $\mathcal{H}_{0}$ denotes the set of true null hypotheses. If we can
keep $\text{FDP}^{*}(t)\leq\alpha$ at all times $t$, then (depending on
dependence assumption on the $p$-values) we can prove that
$\text{mFDR}(t)\leq\alpha$ or $\text{FDR}(t)\leq\alpha$. This technique has
been used to derive the LORD, SAFFRON and ADDIS algorithms (see below), by
designing different estimates
$\widehat{\text{FDP}}_{\text{LORD}}(t),\widehat{\text{FDP}}_{\text{SAFFRON}}(t),\widehat{\text{FDP}}_{\text{ADDIS}}(t)$
for $\text{FDP}^{*}(t)$.

#### LORD

The LORD algorithm was conceptualised by Javanmard and Montanari [2018], and
is an instance of a GAI. Following that, Ramdas et al. [2017] defined a simple
upper bound of $\text{FDP}^{*}(t)$:

$\widehat{\text{FDP}}_{\text{LORD}}(t)=\frac{\sum_{j\leq
t}\alpha_{t}}{R(t)\vee 1},$

and showed that LORD can be viewed as an algorithm that keeps
$\widehat{\text{FDP}}_{\text{LORD}}(t)\leq\alpha$. Exploiting this view, they
derived a uniform improvement of LORD, termed LORD++, presented next. Given an
infinite non-increasing sequence of positive constants
$\\{\gamma_{t}\\}_{t=1}^{\infty}$ that sums to one, the test levels
$\alpha_{t}$ for LORD++ are chosen as follows:

$\alpha_{t}=w_{0}\gamma_{t}+(\alpha-
w_{0})\gamma_{t-\tau_{1}}1_{\tau_{1}<t}+\alpha\sum_{j\,:\,\tau_{j}<t,\,\tau_{j}\neq\tau_{1}}\gamma_{t-\tau_{j}},$

where $\tau_{j}$ denotes the time of the $j$-th rejection. The above formula
may look daunting but it is interpretable. The first term is the fraction of
the initial wealth $w_{0}$ that is used by the $t$-th test. The other terms
are the fractions of the earnings from rejections before $t$ that are spent in
the $t$-th round: LORD++ awards $\alpha-w_{0}$ for the first rejection and
$\alpha$ for every subsequent rejection, and on receiving this reward, the
method immediately allocates that reward to future rounds according to the
same schedule of constants $\\{\gamma_{t}\\}$, shifted to start at the next
instant. This rule ensures that LORD++ never spends more than it has earned,
thus keeping $\widehat{\text{FDP}}_{\text{LORD}}(t)\leq\alpha$.

The manner in which $\widehat{\text{FDP}}_{\text{LORD}}(t)$ is a simple upper
bound on $\text{FDP}^{*}(t)$ is reminiscent of the seminal Benjamini-Hochberg
(BH) procedure, which can be derived in a similar fashion. This leads us to
view LORD++ as the online analog of the BH procedure.

Guarantees for LORD hold under different $p$-value dependencies, which we now
formalise. Define the filtration at time $t$ as
$\mathcal{F}^{t}=\sigma(R_{1},\ldots,R_{t})$ and let
$\alpha_{t}=f_{t}(R_{1},\ldots,R_{t-1})$ where $f_{t}$ is an arbitrary
$[0,1]$-valued function. The null $p$-values are said to be conditionally
super-uniform if
$\text{Pr}\\{P_{t}\leq\alpha_{t}|\mathcal{F}^{t-1}\\}\leq\alpha_{t}$ for any
$\mathcal{F}^{t-1}$-measurable $\alpha_{t}$. Armed with this definition, we
have the following theorem from Ramdas et al. [2017]:

######  Theorem 2.2.

(a) If the null $p$-values are conditionally super-uniform, then the condition
$\widehat{\text{FDP}}_{\text{LORD}}(t)\leq\alpha$ for all $t\geq 1$ implies
that $\text{mFDR}(t)\leq\alpha$ for all $t\geq 1$.
(b) If the null $p$-values are independent of each other and of the non-nulls,
and $\\{\alpha_{t}\\}$ is a monotone function of past rejections, then the
condition $\widehat{\text{FDP}}_{\text{LORD}}(t)\leq\alpha$ for all $t\geq 1$
implies that $\text{FDR}(t)\leq\alpha$ for all $t\geq 1$.

Finally, in terms of theoretical power guarantees for LORD, Chen and Arias-
Castro [2021] considered the setting of a (generalised) Gaussian model and
showed that LORD is asymptotically optimal, in particular by being as powerful
as BH to first asymptotic order.

#### SAFFRON

Ramdas et al. [2018] derived an adaptive version of LORD++ called SAFFRON,
which is based on an estimate of the proportion of true null hypotheses. By
not wasting its earnings on attempting to reject weaker signals (i.e. larger
$p$-values), SAFFRON preserves alpha-wealth and hence can have a higher power
than LORD++.

To this end, choose $\lambda\in(0,1)$ and define the candidate $p$-values as
those that satisfy $P_{t}\leq\lambda$, since SAFFRON will never reject a
$p$-value larger than $\lambda$. For the extension to a sequence
$\\{\lambda_{t}\\}_{t=1}^{\infty}$, see Ramdas et al. [2018]. SAFFRON then
proceeds as follows:

  1. 1.

At each time $t$, define the number of candidates after the $j$-th rejection
as

$C_{j+}=C_{j+}(t)=\sum_{i=\tau_{j}+1}^{t-1}C_{i},$

where $C_{t}=\mathbbm{1}\\{P_{t}\leq\lambda\\}$.

  2. 2.

SAFFRON starts with $\alpha_{1}=\min\\{(1-\lambda)\gamma_{1}w_{0},\lambda\\}$.
Subsequent levels are chosen as
$\alpha_{t}=\min\\{\lambda,\tilde{\alpha}_{t}\\}$, where

$\begin{split}\tilde{\alpha}_{t}=(1-\lambda)[w_{0}\gamma_{t-C_{0+}}+(\alpha-
w_{0})\gamma_{t-\tau_{1}-C_{1+}}+\\\ \alpha\sum_{j\geq
2}\,\gamma_{t-\tau_{j}-C_{j+}}].\end{split}$

Hence, SAFFRON starts off with alpha-wealth $(1-\lambda)w_{0}$ and does not
lose any of this wealth when testing candidate $p$-values. It gains an alpha-
wealth of $(1-\lambda)\alpha$ for each discovery after the first. SAFFRON can
make more rejections than LORD if there is a significant fraction of non-nulls
and the signals are strong.

Similarly to LORD, SAFFRON provably controls the mFDR at all times if the null
$p$-values are conditionally super-uniformly distributed. SAFFRON provably
controls the FDR at all times if the null $p$-values are independent of each
other and of the non-nulls, and $\\{\alpha_{t}\\}$ is a monotone function of
$(R_{1},\ldots,R_{t-1},C_{1},\ldots,C_{t-1})$; see Ramdas et al. [2018] for
full details.

#### ADDIS

ADDIS stands for an ADaptive algorithm that DIScards conservative nulls, and
was proposed by Tian and Ramdas [2019]. ADDIS can invest alpha-wealth more
effectively than LORD or SAFFRON by explicitly discarding the weakest signals
(i.e. the largest $p$-values) in a principled way, which can lead to a higher
power. More formally, in practice it is common to encounter conservative
nulls, where a null $p$-value $P$ is conservative if $\text{Pr}\\{P\leq
x\\}<x$ for all $x\in[0,1]$. Often nulls are uniformly conservative, which
means that under the null,

$\text{Pr}\\{P/c\leq x\,|\,P\leq c\\}\leq x\quad\text{for all }x,c\in(0,1).$

For example, for a one-dimensional exponential family with parameter $\theta$,
when the true parameter $\theta$ is strictly smaller than $\theta_{0}$, the
uniformly most powerful test of $H_{0}:\theta\leq\theta_{0}$ versus
$H_{1}:\theta>\theta_{0}$ will give uniformly conservative nulls [Zhao et al.,
2019]. Another setting is using always-valid $p$-values [Johari et al., 2021]
in the context of continuous monitoring for A/B testing, which will always be
conservative.

With $R_{t}$, $C_{t}$ and $C_{j+}$ defined as for SAFFRON, we now also define
$S_{t}=\mathbbm{1}\\{P_{t}\leq\eta\\}$ as the indicator of $H_{t}$ being
selected for testing (i.e. not discarded). Hence $\eta$ is the discarding
threshold, which can also be extended to a sequence
$\\{\eta_{t}\\}_{t=1}^{\infty}$, see Tian and Ramdas [2019]. We also choose an
infinite non-increasing sequence of positive constants
$\\{\gamma_{t}\\}_{t=0}^{\infty}$ that sums to one. The testing levels for
ADDIS are then given by $\alpha_{t}=\min\\{\lambda,\hat{\alpha}_{t}\\}$, where

$\begin{split}\hat{\alpha}_{t}=(\eta-\lambda)[w_{0}\gamma_{S^{t}-C_{0+}}+(\alpha-
w_{0})\gamma_{S^{t}-\tau_{1}^{*}-C_{1+}}+\\\ \alpha\sum_{j\geq
2}\,\gamma_{S^{t}-\tau_{j}^{*}-C_{j+}}]\end{split}$

and $S^{t}=\sum_{i<t}\mathbbm{1}\\{P_{i}\leq\eta\\}$,
$\tau_{j}^{*}=\sum_{i\leq\tau_{j}}\mathbbm{1}\\{P_{i}\leq\eta\\}$.
See Tian and Ramdas [2019] for an alternative formulation of ADDIS where
$p$-values greater than $\eta$ are explicitly discarded.

Like for LORD and SAFFRON, ADDIS provably controls the mFDR at all times if
the null $p$-values are conditionally uniformly conservative. ADDIS provably
controls the FDR at all times if the null $p$-values are independent of each
other and of the non-nulls, and $\\{\alpha_{t}\\}_{t=1}^{\infty}$ is a
monotone function of the past; see Tian and Ramdas [2019] for full details.

##  3 Simulation studies

In this section, we compare the performance of the LORD, SAFFRON and ADDIS
algorithms in terms of the FDR and statistical power. For reasons of space, we
do not aim to give a comprehensive set of simulations but instead focus on a
few key settings to give a flavour of typical results. In Section 3.3 we refer
the reader to further simulation settings that have been published in the
literature. First though, we briefly describe software implementation of
algorithms for online error rate control.

###  3.1 Software: onlineFDR package

The onlineFDR package is an open-source R package that aims to provide a
comprehensive and up-to-date implementation of algorithms for online error
rate control. It is freely available via Bioconductor [Robertson et al.,
2021]. The package implements the LORD, SAFFRON and ADDIS algorithms, as well
as almost all of the algorithms corresponding to the further extensions of
online error rate control methodology (see Section 5.1). In particular, it
also provides functions for algorithms for online FWER and online FDX control.
The package documentation provides a user-friendly introduction to the use of
the package, and there is also a Shiny app available [Liou and Robertson,
2021] to allow users to explore algorithms for online FDR control in an
interactive way without having to program. All results for the simulation
studies and case studies in this paper were calculated using the package.

###  3.2 Testing with Gaussian observations

In order to examine the relative performance of the online FDR algorithms, we
use a simple experimental setup of testing Gaussian means, with a total of $T$
hypotheses. The null hypotheses take the form $H_{t}:\mu_{t}\leq 0$ which are
tested against the alternative $H_{t}^{\prime}:\mu_{t}>0$ for $t=1,\ldots,T$.
We observe independent observations $Z_{t}\sim N(\mu_{t},1)$ which are
transformed to one-sided $p$-values $P_{t}=\Phi(-Z_{t})$, where $\Phi$ denotes
the standard Gaussian CDF. The means $\mu_{t}$ are set according to the
following mixture model:

$\mu_{t}=\begin{cases}F_{0}\qquad\text{with probability }1-\pi_{1}\\\
F_{1}\qquad\text{with probability }\pi_{1}\end{cases},$

where $F_{1}\sim N(3,1)$ and $F_{0}$ is defined as below.

We use the default settings for LORD, SAFFRON and ADDIS that have been
suggested in the literature. For LORD, we choose the sequence
$\gamma_{t}\propto\frac{\log(t\vee 2)}{t\,\text{exp}(\sqrt{\log t})}$, which
has been shown to maximise its power in the Gaussian setting [Javanmard and
Montanari, 2018]. We also use this choice of $\gamma_{t}$ for alpha-spending,
where $\\{\alpha_{t}\\}_{t=1}^{\infty}$ is simply given by
$\alpha_{t}=\alpha\gamma_{t}$. For SAFFRON we set $\lambda=0.5$ and
$\gamma_{t}\propto\frac{1}{t^{1.6}}$. Finally, for ADDIS we set
$\lambda=0.25$, $\eta=0.5$ and $\gamma_{t}\propto\frac{1}{(t+1)^{1.6}}$.

Figure 2 shows how the test levels $\\{\alpha_{t}\\}_{t=1}^{\infty}$
(displayed on the $\log_{10}$ scale) evolve over time for LORD, SAFFRON and
ADDIS compared with uncorrected testing (where $\alpha_{t}\equiv\alpha$) and
alpha-spending. Here, $T=300$, $\alpha=0.05$, $\pi_{1}=0.5$ and we choose
$F_{0}\equiv 0$.

Figure 2: Test levels for LORD, SAFFRON and ADDIS compared with uncorrected
testing and alpha-spending. We set $T=300$, $\alpha=0.05$ and the proportion
of non-nulls $\pi_{1}=0.5$.

All of the online FDR algorithms have higher test levels than alpha-spending
(apart from LORD briefly early on in this particular experiment). The relative
difference increases with $t$ as the online algorithms ‘earn back’ wealth over
time, which alpha-spending cannot do. Both SAFFRON and ADDIS have higher test
levels than LORD, reflecting how they can more efficiently invest the alpha-
wealth. In this setting, since $\mu_{t}=0$ under the null, the nulls are
exactly uniform and so ADDIS cannot take advantage of conservative nulls.
Hence the testing levels of ADDIS are similar or slightly lower than those for
SAFFRON. Finally, we see that SAFFRON has similar test levels as uncorrected
testing, and both SAFFRON and ADDIS can even have test levels above the
nominal $\alpha$.

Figure 3 compares the statistical power of LORD, SAFFRON and ADDIS compared
with uncorrected testing and alpha-spending, as $\pi_{1}$ varies from 0.01 to
0.9. Here, we define power as

$\text{power}(T)=\mathbb{E}\left[\frac{\sum_{t\in\mathcal{H}_{1}}R_{t}}{\left(\sum_{t=1}^{T}\mathbbm{1}\\{t\in\mathcal{H}_{1}\\}\right)\vee
1}\right],$

where $\mathcal{H}_{1}$ denotes the index set of the non-null hypotheses. We
also include the standard Benjamini-Hochberg (BH) procedure as an additional
comparison. We stress that BH is an offline procedure and so could not be used
for online testing in practice. In our simulation, we set $T=1000$,
$\alpha=0.05$ and $F_{0}\sim N(-0.5,0.1)$. Results are based on averaging
$10^{4}$ simulation replicates.

Figure 3: Power of LORD, SAFFRON and ADDIS compared with uncorrected testing,
the BH procedure and alpha-spending as the proportion of non-nulls $\pi_{1}$
varies. We set $T=1000$ and $\alpha=0.05$. Results are based on $10^{4}$
simulation replicates.

Starting with alpha-spending, as expected the power is very low ($<0.2$) for
all $\pi_{1}$. LORD has substantial power gains compared with alpha-spending
(as long as $\pi_{1}$ is not close to zero) and this advantage increases with
$\pi_{1}$. However, LORD has substantially lower power than BH for all values
of $\pi_{1}$. As expected, SAFFRON performs better as the fraction of non-
nulls $\pi_{1}$ increases, with a higher power than LORD for $\pi_{1}>0.05$,
BH for $\pi_{1}>0.5$ and even uncorrected testing for $\pi_{1}>0.7$. Since
$F_{0}\sim N(-0.5,0.1)$, almost all the means for the null hypotheses will be
negative, i.e. we are in a setting with conservative nulls. Hence, as
expected, ADDIS outperforms SAFFRON in terms of power (except for very high
values of $\pi_{1}$). ADDIS also has a higher power than BH for $\pi_{1}>0.2$
and uncorrected testing for $\pi_{1}>0.6$.

In the Appendix (Figure 5) we show the corresponding FDR for all of the
algorithms considered. We see that uncorrected testing can have substantial
inflation of the FDR, with the FDR inflated above the nominal $\alpha=0.05$
level for $\pi_{1}<0.3$. The FDR reaches as high as 0.65 when $\pi_{1}=0.01$.
All other algorithms control the FDR below the nominal 0.05 level, as
expected.

###  3.3 Observations from other simulations

Here, we summarise a few take-home messages for LORD, SAFFRON and ADDIS from
simulation results already found in the literature. Javanmard and Montanari
[2018] investigated the effect of the ordering of the hypotheses for online
testing rules, including LORD. In some applications, hypotheses can be ordered
using side information, such that those that are most likely to be rejected
come first. With this favourable ordering, the statistical power of LORD can
substantially increase as long as $\pi_{1}$ is not too large (since ordering
is less relevant in that case). Similar findings for LORD, SAFFRON and ADDIS
in the context of platform trials can be found in Robertson et al. [2022],
which also looked at the adversarial setting where hypotheses happen to be
ordered so that those most likely to be rejected come last, resulting in a
lower statistical power.

Ramdas et al. [2018] considered the impact on LORD and SAFFRON of choosing
sequences of the form $\gamma_{t}\propto t^{-s}$, where the parameter $s>1$
controls the ‘aggressiveness’ of the procedure (since the larger the value of
$s$, the more the alpha-wealth is concentrated at small values of $t$). For
Gaussian alternatives, the simulation results suggested that less aggressive
sequences are to be preferred in terms of increased power for SAFFRON and
LORD. Meanwhile, Tian and Ramdas [2019] showed that ADDIS can match the power
of SAFFRON when the nulls are not conservative (i.e. uniform nulls). The power
advantage of ADDIS over LORD and SAFFRON increases the more conservative the
nulls are, i.e. the more negative the means for the null hypotheses are (in
the Gaussian setting).

The theory presented in Section 2 for provable FDR control requires null
$p$-values to be independent of one another (with a weaker condition sufficing
for mFDR control). Robertson and Wason [2018] explored the performance of
online testing rules, including LORD and SAFFRON, when $p$-values are
correlated with each other. In particular, the test statistics are assumed to
come from a multivariate normal distribution where the covariance matrix has
ones along the diagonals and off-diagonal entries equal to $\pm 0.5$. There
was no evidence of FDR inflation when using LORD under a range of non-null
distributions, showing empirically that it is robust to reasonable departures
from independence. However, with a two-sided test under a Gaussian
alternative, the SAFFRON procedure had an inflated FDR for smaller values of
$\pi_{1}$. This inflation persisted and even increased as $T$ increased from
100 to 1000.

More generally, the simulation studies in Robertson and Wason [2018] also
highlighted the value of using ‘bounded’ versions of online testing
algorithms. This requires setting an a-priori upper bound $M$ on the number of
hypotheses to be tested, so that the $\gamma_{t}\equiv 0$ for $t>M$, which
allows setting $\gamma_{t}\equiv 1/M$ for $t\leq M$ (for example). The bounded
versions have a uniformly higher power than the versions presented in Sections
2 (with the default choices of $\gamma_{t}$ given in Section 3.2) which assume
no upper bound on $T$, and empirically a substantial gain can be observed for
small $T$ (i.e. $T<100$). Finally, another general observation is that the
power advantages of online testing algorithms compared with alpha-spending
increase as $T$ increases. Indeed, when $T$ is small and $\pi_{1}$ is low,
online testing algorithms may no longer be competitive in terms of power. We
return to this issue in Section 5.2.

##  4 Case studies

###  4.1 IMPC dataset

Our first case study uses high-throughput phenotypic data from the
International Mouse Phenotyping Consortium (IMPC) data repository, which aims
to generate and phenotypically characterize knockout mutant strains for every
protein-coding gene in the mouse [Koscielny et al., 2013]. The IMPC database
is an example of a growing dataset mentioned in Section 1.1, since the family
of hypotheses is constantly growing as new knockout mice lines are generated
and phenotyping data is uploaded to the data repository.

We focus on the analysis of IMPC data performed by Karp et al. [2017], who
looked at the influence of sex in mammalian phenotypic traits in both wildtype
and mutants. As part of their analysis, Karp et al. analysed the role of sex
as a modifier of the genotype effect (for continuous traits) using a two stage
pipeline. Stage 1 tested the role of genotype using a likelihood ratio test
comparing models (a) and (c). Similarly, stage 2 tested the role of sex using
a likelihood ratio test comparing models (a) and (b).

$\displaystyle(a)\quad Y$
$\displaystyle\sim\text{Genotype}+\text{Sex}+\text{Genotype}*\text{Sex}+\text{Weight}+(1|\text{Batch})$
$\displaystyle(b)\quad Y$
$\displaystyle\sim\text{Genotype}+\text{Sex}+\text{Weight}+(1|\text{Batch})$
$\displaystyle(c)\quad Y$
$\displaystyle\sim\text{Sex}+\text{Weight}+(1|\text{Batch})$

The above procedure resulted in two sets of $N=172\,328$ distinct $p$-values.
Note that these $p$-values will not be independent, due to positive and
negative associations between different genes (caused for instance by linkage
disequilibrium). In addition, multiple variables are being measured for the
same gene, and these can be aspects of the same phenotype or be biologically
correlated.

Table 1 below shows the number of traits that had a statistically significant
genotype effect or were classed as having a statistically significant sexual
dimorphism (SD) using LORD, SAFFRON and ADDIS. As a comparison, we include the
results from alpha-spending, BH and uncorrected testing. The ‘Fixed’ procedure
is the fixed significance threshold of 0.0001 used in practice for the IMPC
pipeline.

|  Genotype |  SD
---|---|---
Uncorrected |  35 575 |  20 887
BH |  12 907 |  2 084
ADDIS |  12 558 |  1 713
SAFFRON |  14 268 |  1 705
LORD |  8 517 |  1 193
Fixed |  4 158 |  969
Alpha-spending |  795 |  60
Table 1: Number of rejections made by online FDR algorithms and various
comparators using the IMPC datasets. SD = Sexual Dimorphism.

Starting first with the results for the genotype data, the online testing
algorithms make two to three times as many rejections as fixed testing. ADDIS
and SAFFRON in turn make substantially more rejections than LORD, with an
increase of almost 50% and 70%, respectively. ADDIS makes a similar number of
rejections to BH, but SAFFRON makes noticeably more rejections than both BH
and ADDIS for these data. For the SD data, again the online testing algorithms
make substantially more rejections than fixed testing, but the relative
increase is much less. ADDIS and SAFFRON make a very similar number of
rejections, about 50% more than the number of rejections for LORD.

###  4.2 Platform trial: STAMPEDE

Our second case study is the ongoing STAMPEDE (Systemic Therapy for Advancing
or Metastatic Prostate Cancer) platform trial, which evaluates the effect of
systemic therapies for prostate cancer on overall survival [James et al.,
2008]. The trial started with 5 experimental treatment arms (B – F), and
compared these with the control arm A, which was standard-of-care (SOC)
hormone therapy. Figure 4 shows a schematic of the treatment comparisons that
have already been reported from STAMPEDE. Two additional experimental arms (G
and H) were added to the trial in 2011 and 2013, respectively.

Figure 4: Schematic of the completed treatment arms in the STAMPEDE platform
trial. ab = abiraterone, rt = radiotherapy.

Table 2 shows the reported $p$-values (unadjusted for multiplicity) when
comparing the experimental arms with the control (arm A), as given in James et
al. [2016, 2017], Mason et al. [2017], Parker et al. [2018]. The dashed lines
denote the four ‘batches’ present in the trial, where a batch corresponds to
multiple hypotheses being available to be tested at the same time, as
reflected in Figure 4.

Trial arm |  $p$-value
---|---
B: SOC + zoledronic acid | 0.450
C: SOC + docetaxel | 0.006
E: SOC + zoledronic acid + docetaxel | 0.022
D: SOC + celecoxib | 0.847
F: SOC + zoledronic acid + celecoxib | 0.130
G: SOC + abiraterone | 0.001
H: SOC + radiotherapy | 0.266
Table 2: Reported $p$-values for the STAMPEDE platform trial. SOC = Standard-
of-care.

Following Robertson et al. [2022], we apply the online testing algorithms to
these observed $p$-values, keeping the alphabetical ordering of $p$-values
within the batches. We set the upper bound on the number of treatments $M=20$
(i.e. twice as many arms that have already entered the STAMPEDE trial as of
the end of 2021), and use the bounded versions of alpha-spending (i.e. a
Bonferroni correction at level $\alpha/M$), LORD, SAFFRON and ADDIS. Table 3
shows which of the hypotheses corresponding to each experimental arm can be
rejected at level $\alpha=0.05$, as well as the current significance level
$\alpha_{8}$ that would be used to test the next experimental treatment after
the 7 already evaluated in the trial.

Algorithm | Rejections | $\bm{\alpha_{8}}$
---|---|---
Uncorrected | C, E, G | 0.0500
Alpha-spending | G | 0.0025
BH | C, G | –
ADDIS | G | 0.0016
SAFFRON | C, G | 0.0165
LORD | – | 0.0002
Table 3: Hypotheses rejected and current significance level $\alpha_{8}$ of
different algorithms using the results of the STAMPEDE trial, with the
ordering as in Table 2.

Uncorrected testing rejects the hypotheses corresponding to three experimental
arms (C, E, G), and has by far the highest value of $\alpha_{8}$. Both SAFFRON
and the BH procedure reject hypotheses C and G, and SAFFRON has a
substantially higher value of $\alpha_{8}$ than for the other online testing
algorithms. ADDIS and alpha-spending only reject hypothesis G, and have
similar $\alpha_{8}$. Finally, LORD does not reject any hypotheses and the
value of $\alpha_{8}$ is substantially lower than any of the other algorithms.
For further discussion and results, see Robertson et al. [2022].

##  5 Extensions and future directions

###  5.1 Further extensions

Prior weights, penalty weights and decaying memory
Ramdas et al. [2017] proposed a number of extensions that apply to the class
of GAI++ algorithms, including LORD++. Firstly, they showed how to incorporate
certain types of prior information about the different hypotheses as expressed
through prior weights $w_{t}$ and penalty weights $u_{t}$. Prior weights allow
the experimenter to exploit domain knowledge about which hypotheses are more
likely to be non-null. By assigning a higher prior weight $w_{t}>1$ to a
hypothesis, the algorithm will have a higher chance of rejecting $H_{t}$.
Meanwhile, penalty weights express the different importance attached to the
hypotheses being tested, with $u_{t}>1$ indicating a more impactful or
important test. Importantly, both $w_{t}$ and $u_{t}$ are allowed to depend on
past rejections in this framework. Ramdas et al. [2017] proposed doubly-
weighted GAI++ rules that provably control the penalty-weighted FDR when using
both prior and penalty weights under independence.

The second proposal dealt with problems of ‘piggybacking’ and ‘alpha-death’.
Piggybacking happens when a substantial number of rejections are made so that
the online testing algorithms earn and accumulate enough alpha-wealth to
reject later hypotheses at much less stringent thresholds (hence the later
tests ‘piggyback’ on the success of earlier tests). This can lead to a spike
in the FDR locally in time, even though the FDR over all time is controlled.
Meanwhile, alpha-death occurs when there is a long stretch of null hypotheses,
so that online testing algorithms make (almost) no rejections and lose nearly
all of their alpha-wealth. Subsequently, the algorithm may have essentially no
power, unless a non-null hypotheses with extremely strong signal (small
$p$-value) is observed. Ramdas et al. [2017] proposed the decaying memory FDR
(mem-FDR), which pays more attention to recent discoveries through a user-
defined discount factor $\delta\in(0,1]$ and thus smoothly forgets the past.
They then proposed GAI++ rules that control the mem-FDR (which can also
include penalty weights) under independence. In addition, they showed how to
allow the algorithm to abstain from testing in order to recover from alpha-
death.

Asynchronous and batched testing
In most of the literature mentioned so far, an implicit assumption is that
each hypothesis test can only start when the previous test has finished (the
synchronous setting). In reality, experimentation is “doubly sequential” like
in Figure 1, where it is common to have hypothesis tests that overlap in time,
where each test may itself be run sequentially (the asynchronous setting). One
natural adjustment for asynchronous testing is to use an online FDR algorithm
whenever each test finishes (that is, whichever test is the $t$-th one to
finish, test it at level $\alpha_{t}$). However, this would only assign
$\alpha_{t}$ at the end of a hypothesis test, which would not be appropriate
for sequential hypothesis testing and multi-arm bandit approaches that
typically require specification of the target type I error level in advance
because it is an important component of their stopping rule. Hence, the
testing levels must be specified at the start of a hypothesis test. The
asynchronous setting also means that potentially arbitrary dependence between
some $p$-values must be considered. Indeed, hypothesis tests that are being
conducted concurrently are often likely to be dependent, since they may use
the same or highly correlated data during their overlap.

To address these challenges of asynchronous testing, Zrnic et al. [2021]
derived asynchronous versions of LORD and SAFFRON that output test levels
$\alpha_{t}$ dynamically at the beginning of the $t$-th test, such that,
despite arbitrary local dependence and regardless of the decision times for
each hypothesis, the mFDR is controlled at level $\alpha$. These procedures
achieve this goal both at all fixed times $t$, as well as certain adaptively
chosen stopping times. Tian and Ramdas [2019] also showed how to derive
asynchronous versions of ADDIS. Note that in order to account for the
uncertainty about the tests in progress, the test levels assigned by
asynchronous online procedures must be more conservative. Thus, there is a
trade-off in that although asynchronous procedures take less time to perform a
given number of tests, they can be less powerful than their synchronous
counterparts.

Zrnic et al. [2020] considered the related setting of online batched testing,
where a potentially infinite number of batches of hypotheses are tested over
time (see Section 4.2 for an example). To this end, they introduced online,
FDR-preserving versions of the most widely used offline algorithms, namely the
BH procedure and Storey’s improvement of the BH method [Storey, 2002]. These
online “mini-batched” testing algorithms interpolate between online and
offline methodology, thus trading off the best of both worlds. These
algorithms control the FDR under independence (an algorithm valid under
positive dependence was also derived), and have a higher power than the fully
online testing algorithms. Further, since they consist of compositions of
offline FDR algorithms, they imply FDR control over each constituent batch,
and not just over the whole sequence of tests.

A Bayesian approach
Gang et al. [2021] developed a new class of structure-adaptive sequential
testing (SAST) rules for online FDR control, which instead of being based on
$p$-values, are based upon estimates of the conditional local FDR (Clfdr; Cai
and Sun [2009]), which can optimally adapt to important local structures in
the data stream. This results in a novel alpha-investing framework that more
precisely characterises the effects of rejection of hypotheses: rather than
viewing each rejection as a gain of alpha-wealth, the Clfdr characterisation
does not view all rejections as equal. Rejections with small Clfdr will lead
to increased alpha-wealth whereas rejections with large Clfdr will lead to
decreased alpha-wealth. SAST learns the optimal rejection thresholds
adaptively and optimises the alpha-wealth allocation across different time
periods.

Gang et al. [2021] showed that SAST can achieve substantial power gain over
existing methods, but it comes at the cost of only asymptotically controlling
the FDR if the underlying Bayesian model is well-specified.

Online FWER control
Tian and Ramdas [2021] focused on developing methods for online control of the
FWER (see Section 1.2). Starting with the observation that only alpha-spending
had previously been proposed for online FWER control, the authors first
extended existing offline algorithms for FWER control, namely the Sidak method
[Šidák, 1967] and the fallback procedure [Burman et al., 2009]. The online
Sidak and online fallback procedures then control the FWER at all times $t$
under independence and arbitrary dependence, respectively. However, although
these online FWER control methods are guaranteed to be uniformly more powerful
than alpha-spending, the improvements are usually minor in practice, except
for extreme cases. Hence, Tian and Ramdas [2021] proposed the ADDIS-spending
algorithm for online FWER control, which (like ADDIS does for online FDR
control) benefits from adaptivity to the fraction of nulls, but also gains
power by discarding conservative nulls (if they exist). ADDIS-spending
controls the FWER when null $p$-values are uniformly conservative, and
independent of each other and of the non-nulls.

Online FDX control and FDR at stopping times
Xu and Ramdas [2022] focused on online control of the FDX (see Section 1.2).
Prior to this work, the only online procedure that controlled the FDX was
proposed by Javanmard and Montanari [2018], but this had very low power (no
better than alpha-spending). Xu and Ramdas [2022] proposed the supLORD
algorithm, which has a higher power and provably controls the FDX when the
null $p$-values are conditionally superuniform. One feature of this algorithm
is that it allows the user to choose the number of rejections after which FDX
control begins, in exchange for more power. SupLORD is based on the GAI
framework, and the authors also show how to dynamically chooses larger test
levels $\alpha_{t}$ when the wealth is large, allowing the algorithm to fully
utilise its wealth and increases its power as a result. Finally, the authors
show that supLORD also controls the mFDR and FDR at both fixed times and
stopping times. Hence, supLORD provides the first guarantee for online FDR
control at stopping times (LORD, SAFFRON and ADDIS only control the mFDR at
stopping times and not the FDR).

Retesting of hypotheses
One feature of online testing algorithms that has not been explicitly pointed
out in the literature is the option of retesting hypotheses (i.e. using the
same $p$-value again later in the testing sequence, when the alpha-wealth may
be higher). Crucially however, the choice of whether to retest must be made
without using knowledge of the $p$-value itself, but only that it was not
rejected (e.g. that it is greater 0.01). In this example, under the null the
$p$-value will still be conditionally uniform in $[0.01,1]$. Since this
implies that the assumption of conditional super-uniformity under the null
still holds, the same $p$-value can be used for retesting. In practice,
retesting could happen within an automated testing setting for example,
perhaps with additional prior information. Meanwhile, Fisher [2022] recently
proposed a framework for online testing where each hypothesis requires an
immediate preliminary decision, which allows the analyst to update that
decision until a preset deadline while still controlling the FDR.

Discrete test statistics
Döhler et al. [2021] focused on the setting where the null $p$-values are
conservative due to the discreteness of the test statistics, i.e. where the
individual tests are based on counts or contingency tables. The authors
proposed uniform improvements of LORD, SAFFRON and ADDIS-spending, and showed
that the power gains can be substantial when the discreteness is high (i.e.
the counts in the contingency are moderate).

Post-hoc FDP bounds
Katsevich and Ramdas [2020] proposed a class of simultaneous FDP bounds that
apply to a variety of settings, including online testing. These bounds are
finite-sample have a simple closed form. The results can be used as a
diagnostic tool for FDR procedures: after running an FDR procedure, one can
obtain a valid $1-\alpha$ confidence bound on the FDP of the resulting
rejection set. Since the guarantees are post hoc, they apply to any sequence
of rejections produced by any online algorithm, that may or may not have been
designed for FDR or FDP control.

Online control of the False Coverage Rate
Finally, Weinstein and Ramdas [2020] considered the problem of constructing
confidence intervals (CIs) that are valid for online hypothesis testing. In
particular, they focus on control of the false coverage rate (FCR), which is
the expected ratio of number of constructed CIs that fail to cover their
respective parameters to the total number of constructed CIs. In the online
hypothesis testing framework they considered, at each step the investigator
observes independent data that are informative about the parameter of interest
$\theta_{t}$, and must immediately make a decision whether to report a CI for
$\theta_{t}$ or not. If a CI is reported for $\theta_{t}$, then the the aim is
to ensure that that the CI for $\theta_{t}$ has $\text{FCR}\leq\alpha$ at all
times $T$. For further details of the proposed algorithms and their
theoretical guarantees, see Weinstein and Ramdas [2020].

###  5.2 Current shortcomings and future directions

Online testing for small numbers of hypotheses
Online testing algorithms are most powerful in settings where there are a
large number (i.e. $T>1000$) of hypotheses that will eventually be tested.
Thus the biggest advantage will likely be in settings such as A/B testing in
large tech companies or in large-scale biological data repositories. However,
while platform trials provide a framework for a trial to continue indefinitely
in theory, in practice they will typically evaluate a maximum number of
interventions in the low tens. Hence, there is a need for investigation of
optimal online testing procedures when the maximum number of hypotheses to be
tested is relatively low and when the correlation between hypotheses is known
(e.g. because of a shared control arm). Separately, since many platform trials
use an adaptive design, methods that explicitly combine group-sequential
designs with online control of error rates would be useful (see Zehetmayer et
al. [2021] for a recent proposal along these lines).

Managing incentives across sponsors or products
There are some additional challenges in using online control methods in
platform trials or in the IT industry. If different sponsors (e.g.
pharmaceutical companies) are supporting a platform trial, they might be
reluctant to have their intervention be tested at a notably more stringent
level than other sponsors: it may be difficult to reconcile the most powerful
overall procedure not being acceptable to individual sponsors. Similarly, if a
large IT company imposes that experiments run across various products must all
be subjected to oversight in the form of a common online FDR controlling
procedure that acts across products, then it may be hard to convince
individual product teams that their tests must be subject to a level
determined by the results of experiments by other groups.

Optimal choices of parameters for online algorithms
As seen in Section 2, LORD, SAFFRON and ADDIS depend on the choice of the
initial wealth $w_{0}$ as well as the sequence $\\{\gamma_{t}\\}$. Further
work could look at optimal choices of these parameters, given assumptions
about the distribution of non-null $p$-values. Exploring data-adaptive choices
of time varying sequences $\\{\lambda_{t}\\}_{t=1}^{\infty}$ (for SAFFRON and
ADDIS) and $\\{\eta_{t}\\}_{t=1}^{\infty}$ (for ADDIS) with provable power
increase would be another fruitful area of research. Similar future work could
look at optimal choices for the parameters for the other algorithms mentioned
in Section 5.1.

Online batched testing
A number of open questions remain regarding the proposals of Zrnic et al.
[2020] for online batched testing. First, the framework could be extended to
allow for asynchronous online batch testing, using the ideas of Zrnic et al.
[2021]. Second, it should be possible to derive online batched versions based
on the offline counterpart of ADDIS, which would gain power in the presence of
conservative nulls. Third, an open question is determining the trade off
between the chosen batch size versus power in online batched testing.

Online error rate control under dependence
One major shortcoming with almost all of the proposed online testing
algorithms is their reliance on the assumption of independence of the null
$p$-values for provable FDR control, which is unlikely to always be case in
real data applications. However, in terms of online FDR control under
dependence, there have only been limited proposals in the literature. Zrnic et
al. [2021] showed that the LOND algorithm by Javanmard and Montanari [2015]
controls the FDR under positive dependence. In the online setting, arbitrary
dependence of p-values across all time is a rather pessimistic and unrealistic
assumption, and thus in the asynchronous setting, Zrnic et al. [2021]
introduced the concept of arbitrary _local_ dependence and showed that online
algorithms can be modified to control the FDR even with such dependence. See
also Fisher [2021], who showed further results for control of the FDR under
positive dependence in the minibatch setting. Finally, Zrnic et al. [2020]
showed how to control the FDR in the online batched setting under positive
dependence. A useful avenue of future work could explore how to construct
powerful online FDR algorithms under different forms of dependence, including
when the correlation structure is known or can be estimated.

##  6 Discussion

Online error rate control methodology provides a powerful and flexible
framework for large-scale hypothesis testing that takes into account the
temporal nature of modern data analysis. Over the past 15 years since this
framework was first proposed, there have been many proposed improvements and
extensions, which better reflect the nature of real-world data and expand the
scope of potential applications. In particular, continuous progress has been
made towards increasing the statistical power of online testing algorithms, so
that they can match (and in some cases even exceed) the power of traditional
offline algorithms. The issue of accounting for dependent $p$-values remains
open, although progress has been made here too.

As the methodology becomes increasingly mature, the next natural step is to
see application of online testing algorithms in practice. To this end, and as
seen in Section 4, there have been a number of papers that are specifically
focused on application examples, including in the context of growing data
repositories [Robertson et al., 2019], platform trials [Robertson et al.,
2022] and RNAseq data [Liou et al., 2022]. Further work may be required to
explore and solve practical challenges that may arise in different application
settings. Finally, the provision of software and training will also be key to
promoting the use of online error rate control in practice. The onlineFDR
package we described earlier is a key step in that regard, but software tuned
to specific applications may also be desirable.

## Funding

DS Robertson was supported by the Biometrika Trust, the UK Medical Research
Council (MC_UU_0002/14) and the NIHR Cambridge Biomedical Research Centre
(BRC1215-20014). The views expressed in this publication are those of the
authors and not necessarily those of the NHS, the National Institute for
Health Research or the Department of Health and Social Care (DHCS). For the
purpose of open access, the author has applied a Creative Commons Attribution
(CC BY) licence to any Author Accepted Manuscript version arising. JMS Wason
was supported by a NIHR Research Professorship (NIHR301614). A Ramdas was
supported by NSF DMS CAREER award 1916320.

## References

  * 1000 Genomes Project Consortium [2015] 1000 Genomes Project Consortium.  A global reference for human genetic variation.  _Nature_ , 526:68–74, 2015.
  * Aharoni and Rosset [2014] E. Aharoni and S. Rosset.  Generalized $\alpha$-investing: definitions, optimality results and application to public databases.  _Journal of the Royal Statistical Society: Series B (Statistical Methodology)_ , 76(4):771–794, 2014.
  * Berman and Van den Bulte [2021] R. Berman and C. Van den Bulte.  False discovery in A/B testing.  _Management Science_ , 2021.
  * Burman et al. [2009] C.-F. Burman, C. Sonesson, and O. Guilbaud.  A recycling framework for the construction of bonferroni-based multiple tests.  _Statistics in Medicine_ , 28(5):739–761, 2009\.
  * Cai and Sun [2009] T. T. Cai and W. Sun.  Simultaneous testing of grouped hypotheses: Finding needles in multiple haystacks.  _Journal of the American Statistical Association_ , 104(488):1467–1481, 2009.
  * Chen and Arias-Castro [2021] S. Chen and E. Arias-Castro.  On the power of some sequential multiple testing procedures.  _Annals of the Institute of Statistical Mathematics_ , 73(2):311–336, 2021.
  * Dickerman et al. [2019] B. A. Dickerman, X. García-Albéniz, R. W. Logan, S. Denaxas, and M. A. Hernán.  Avoidable flaws in observational analyses: an application to statins and cancer.  _Nature medicine_ , 25(10):1601–1606, 2019.
  * Dickinson et al. [2016] M. E. Dickinson, A. M. Flenniken, X. Ji, L. Teboul, M. D. Wong, J. K. White, T. F. Meehan, W. J. Weninger, H. Westerberg, H. Adissu, C. N. Baker, L. Bower, J. M. Brown, L. B. Caddle, F. Chiani, D. Clary, J. Cleak, M. J. Daly, J. M. Denegre, B. Doe, M. E. Dolan, S. M. Edie, H. Fuchs, V. Gailus-Durner, A. Galli, A. Gambadoro, J. Gallegos, S. Guo, N. R. Horner, C.-W. Hsu, S. J. Johnson, S. Kalaga, L. C. Keith, L. Lanoue, T. N. Lawson, M. Lek, M. Mark, S. Marschall, J. Mason, M. L. McElwee, S. Newbigging, L. M. J. Nutter, K. A. Peterson, R. Ramirez-Solis, D. J. Rowland, E. Ryder, K. E. Samocha, J. R. Seavitt, M. Selloum, Z. Szoke-Kovacs, M. Tamura, A. G. Trainor, I. Tudose, S. Wakana, J. Warren, O. Wendling, D. B. West, L. Wong, A. Yoshiki, The International Mouse Phenotyping Consortium, W. Wurst, D. G. MacArthur, G. P. Tocchini-Valentini, X. Gao, P. Flicek, A. Bradley, W. C. Skarnes, M. J. Justice, H. E. Parkinson, M. Moore, S. Wells, R. E. Braun, K. L. Svenson, M. H. de Angelis, Y. Herault, T. Mohun, A.-M. Mallon, R. M. Henkelman, S. D. M. Brown, D. J. Adams, K. C. K. Lloyd, C. McKerlie, A. L. Beaudet, M. Bućan, and S. A. Murray.  High-throughput discovery of novel developmental phenotypes.  _Nature_ , 537:508–514, 2016.
  * Döhler et al. [2021] S. Döhler, I. Meah, and E. Roquain.  Online multiple testing with super-uniformity reward.  _arXiv preprint arXiv:2110.01255_ , 2021.
  * Fisher [2021] A. Fisher.  SAFFRON and LORD Ensure Online Control of the False Discovery Rate Under Positive Dependence.  _arXiv preprint arXiv:2110.08161_ , 2021.
  * Fisher [2022] A. J. Fisher.  Online Control of the False Discovery Rate under “Decision Deadlines”.  In _International Conference on Artificial Intelligence and Statistics_ , pages 8340–8359. PMLR, 2022.
  * Foster and Stine [2008] D. P. Foster and R. A. Stine.  $\alpha$-investing: a procedure for sequential control of expected false discoveries.  _Journal of the Royal Statistical Society: Series B (Statistical Methodology)_ , 70(2):429–444, 2008.
  * Gang et al. [2021] B. Gang, W. Sun, and W. Wang.  Structure–Adaptive Sequential Testing for Online False Discovery Rate Control.  _Journal of the American Statistical Association_ , pages 1–14, 2021\.
  * Head et al. [2015] M. L. Head, L. Holman, R. Lanfear, A. T. Kahn, and M. D. Jennions.  The extent and consequences of p-hacking in science.  _PLoS Biology_ , 13(3):e1002106, 2015.
  * Howard et al. [2021] S. R. Howard, A. Ramdas, J. McAuliffe, and J. Sekhon.  Time-uniform, nonparametric, nonasymptotic confidence sequences.  _The Annals of Statistics_ , 49(2):1055–1080, 2021.
  * Ioannidis [2005] J. P. Ioannidis.  Why most published research findings are false.  _PLoS Medicine_ , 2(8):e124, 2005.
  * James et al. [2008] N. James, M. Sydes, N. Clarke, M. Mason, D. Dearnaley, J. Anderson, R. Popert, K. Sanders, R. Morgan, J. Stansfeld, J. Dwyer, J. Masters, and M. Parmar.  Stampede: Systemic therapy for advancing or metastatic prostate cancer – a multi-arm multi-stage randomised controlled trial.  _Clinical Oncology_ , 20(8):577–581, 2008.  ISSN 0936-6555.  doi: https://doi.org/10.1016/j.clon.2008.07.002.  URL https://www.sciencedirect.com/science/article/pii/S0936655508003117.
  * James et al. [2016] N. D. James, M. R. Sydes, N. W. Clarke, M. D. Mason, D. P. Dearnaley, M. R. Spears, A. W. S. Ritchie, C. C. Parker, J. M. Russell, G. Attard, J. de Bono, W. Cross, R. J. Jones, G. Thalmann, C. Amos, D. Matheson, R. Millman, M. Alzouebi, S. Beesley, A. J. Birtle, S. Brock, R. Cathomas, P. Chakraborti, S. Chowdhury, A. Cook, T. Elliott, J. Gale, S. Gibbs, J. D. Graham, J. Hetherington, R. Hughes, R. Laing, F. McKinna, D. B. McLaren, J. M. O’Sullivan, O. Parikh, C. Peedell, A. Protheroe, A. J. Robinson, N. Srihari, R. Srinivasan, J. Staffurth, S. Sundar, S. Tolan, D. Tsang, J. Wagstaff, and M. K. B. Parmar.  Addition of docetaxel, zoledronic acid, or both to first-line long-term hormone therapy in prostate cancer (stampede): survival results from an adaptive, multiarm, multistage, platform randomised controlled trial.  _The Lancet_ , 387(10024):1163–1177, 2016.  ISSN 0140-6736.  doi: https://doi.org/10.1016/S0140-6736(15)01037-5.  URL https://www.sciencedirect.com/science/article/pii/S0140673615010375.
  * James et al. [2017] N. D. James, J. S. de Bono, M. R. Spears, N. W. Clarke, M. D. Mason, D. P. Dearnaley, A. W. Ritchie, C. L. Amos, C. Gilson, R. J. Jones, D. Matheson, R. Millman, G. Attard, S. Chowdhury, W. R. Cross, S. Gillessen, C. C. Parker, J. M. Russell, D. R. Berthold, C. Brawley, F. Adab, S. Aung, A. J. Birtle, J. Bowen, S. Brock, P. Chakraborti, C. Ferguson, J. Gale, E. Gray, M. Hingorani, P. J. Hoskin, J. F. Lester, Z. I. Malik, F. McKinna, N. McPhail, J. Money-Kyrle, J. O’Sullivan, O. Parikh, A. Protheroe, A. Robinson, N. N. Srihari, C. Thomas, J. Wagstaff, J. Wylie, A. Zarkar, M. K. Parmar, and M. R. Sydes.  Abiraterone for prostate cancer not previously treated with hormone therapy.  _New England Journal of Medicine_ , 377(4):338–351, 2017.  doi: 10.1056/NEJMoa1702900.  URL https://doi.org/10.1056/NEJMoa1702900.
  * Javanmard and Montanari [2015] A. Javanmard and A. Montanari.  On online control of false discovery rate.  _arXiv preprint arXiv:1502.06197_ , 2015.
  * Javanmard and Montanari [2018] A. Javanmard and A. Montanari.  Online rules for control of false discovery rate and false discovery exceedance.  _Annals of Statistics_ , 46(2):526–554, 2018\.
  * Johari et al. [2021] R. Johari, P. Koomen, L. Pekelis, and D. Walsh.  Always valid inference: Continuous monitoring of A/B tests.  _Operations Research_ , 2021.
  * Karp et al. [2017] N. A. Karp, J. Mason, A. L. Beaudet, Y. Benjamini, L. Bower, R. E. Braun, S. D. Brown, E. J. Chesler, M. E. Dickinson, A. M. Flenniken, H. Fuchs, M. H. de Angelis, X. Gao, S. Guo, S. Greenaway, R. Heller, Y. Herault, M. J. Justice, N. Kurbatova, C. J. Lelliott, K. C. K. Lloyd, A.-M. Mallon, J. E. Mank, H. Masuya, C. McKerlie, T. F. Meehan, R. F. Mott, S. A. Murray, H. Parkinson, R. Ramirez-Solis, L. Santos, J. R. Seavitt, D. Smedley, T. Sorg, A. O. Speak, K. P. Steel, K. L. Svenson, the International Mouse Phenotyping Consortium, S. Wakana, D. West, S. Wells, H. Westerberg, S. Yaacoby, and J. K. White.  Prevalence of sexual dimorphism in mammalian phenotypic traits.  _Nature Communications_ , 8:15475, 2017.
  * Katsevich and Ramdas [2020] E. Katsevich and A. Ramdas.  Simultaneous high-probability bounds on the false discovery proportion in structured, regression and online settings.  _The Annals of Statistics_ , 48(6):3465–3487, 2020.
  * Kohavi et al. [2020] R. Kohavi, D. Tang, Y. Xu, L. G. Hemkens, and J. Ioannidis.  Online randomized controlled experiments at scale: lessons and extensions to medicine.  _Trials_ , 21(1):1–9, 2020.
  * Koscielny et al. [2013] G. Koscielny, G. Yaikhom, V. Iyer, T. F. Meehan, H. Morgan, J. Atienza-Herrero, A. Blake, C.-K. Chen, R. Easty, A. Di Fenza, T. Fiegel, M. Grifiths, A. Horne, N. A. Karp, N. Kurbatova, J. C. Mason, P. Matthews, D. J. Oakley, A. Qazi, J. Regnart, A. Retha, L. A. Santos, D. J. Sneddon, J. Warren, H. Westerberg, R. J. Wilson, D. G. Melvin, D. Smedley, S. D. M. Brown, P. Flicek, W. C. Skarnes, A.-M. Mallon, and H. Parkinson.  The International Mouse Phenotyping Consortium Web Portal, a unified point of access for knockout mice and related phenotyping data.  _Nucleic Acids Research_ , 42(D1):D802–D809, 2013.
  * Liou and Robertson [2021] L. Liou and D. S. Robertson.  OnlineFDRexplore.  http://shiny.mrc-bsu.cam.ac.uk/apps/onlineFDRexplore/, 2021.  Accessed: 2022-06-15.
  * Liou et al. [2022] L. Liou, M. Hornburg, and D. S. Robertson.  Online FDR Control for RNAseq Data.  _arXiv preprint arXiv:2206.02213_ , 2022.
  * Mason et al. [2017] M. D. Mason, N. W. Clarke, N. D. James, D. P. Dearnaley, M. R. Spears, A. W. Ritchie, G. Attard, W. Cross, R. J. Jones, C. C. Parker, J. M. Russell, G. N. Thalmann, F. Schiavone, E. Cassoly, D. Matheson, R. Millman, C. A. Rentsch, J. Barber, C. Gilson, A. Ibrahim, J. Logue, A. Lydon, A. D. Nikapota, J. M. O’Sullivan, E. Porfiri, A. Protheroe, N. N. Srihari, D. Tsang, J. Wagstaff, J. Wallace, C. Walmsley, M. K. Parmar, M. R. Sydes, and STAMPEDE investigators.  Adding celecoxib with or without zoledronic acid for hormone-naïve prostate cancer: Long-term survival results from an adaptive, multiarm, multistage, platform, randomized controlled trial.  _Journal of Clinical Oncology_ , 35(14):1530–1541, 2017.  doi: 10.1200/JCO.2016.69.0677.  URL https://doi.org/10.1200/JCO.2016.69.0677.
  * Parker et al. [2018] C. C. Parker, N. D. James, C. D. Brawley, N. W. Clarke, A. P. Hoyle, A. Ali, A. W. S. Ritchie, G. Attard, S. Chowdhury, W. Cross, D. P. Dearnaley, S. Gillessen, C. Gilson, R. J. Jones, R. E. Langley, Z. I. Malik, M. D. Mason, D. Matheson, R. Millman, J. M. Russell, G. N. Thalmann, C. L. Amos, R. Alonzi, A. Bahl, A. Birtle, O. Din, H. Douis, C. Eswar, J. Gale, M. R. Gannon, S. Jonnada, S. Khaksar, J. F. Lester, J. M. O’Sullivan, O. A. Parikh, I. D. Pedley, D. M. Pudney, D. J. Sheehan, N. N. Srihari, A. T. H. Tran, M. K. B. Parmar, and M. R. Sydes.  Radiotherapy to the primary tumour for newly diagnosed, metastatic prostate cancer (stampede): a randomised controlled phase 3 trial.  _The Lancet_ , 392(10162):2353–2366, 2018.  ISSN 0140-6736.  doi: https://doi.org/10.1016/S0140-6736(18)32486-3.  URL https://www.sciencedirect.com/science/article/pii/S0140673618324863.
  * Ramdas et al. [2017] A. Ramdas, F. Yang, M. J. Wainwright, and M. I. Jordan.  Online control of the false discovery rate with decaying memory.  In _Advances in Neural Information Processing Systems 30_ , pages 5650–5659. 2017.  URL http://papers.nips.cc/paper/7148-online-control-of-the-false-discovery-rate-with-decaying-memory.pdf.
  * Ramdas et al. [2018] A. Ramdas, T. Zrnic, M. Wainwright, and M. Jordan.  SAFFRON: an adaptive algorithm for online control of the false discovery rate.  In _Proceedings of the 35th International Conference on Machine Learning_ , volume 80 of _Proceedings of Machine Learning Research_ , pages 4286–4294, 2018.  URL http://proceedings.mlr.press/v80/ramdas18a.html.
  * Robertson and Wason [2018] D. S. Robertson and J. M. S. Wason.  Online control of the false discovery rate in biomedical research.  _arXiv preprint arXiv:1809.07292_ , 2018.
  * Robertson et al. [2019] D. S. Robertson, J. Wildenhain, A. Javanmard, and N. A. Karp.  onlineFDR: an R package to control the false discovery rate for growing data repositories.  _Bioinformatics_ , 35(20):4196–4199, 03 2019\.  doi: 10.1093/bioinformatics/btz191.  URL https://doi.org/10.1093/bioinformatics/btz191.
  * Robertson et al. [2021] D. S. Robertson, L. Liou, A. Ramdas, and N. A. Karp.  onlineFDR: Online error rate control.  doi:10.18129/B9.bioc.onlineFDR, 2021.
  * Robertson et al. [2022] D. S. Robertson, J. Wason, F. König, M. Posch, and T. Jaki.  Online error control for platform trials.  _arXiv preprint arXiv:2202.03838_ , 2022.
  * Saville and Berry [2016] B. R. Saville and S. M. Berry.  Efficiencies of platform clinical trials: a vision of the future.  _Clinical Trials_ , 13(3):358–366, 2016.
  * Šidák [1967] Z. Šidák.  Rectangular confidence regions for the means of multivariate normal distributions.  _Journal of the American Statistical Association_ , 62(318):626–633, 1967.
  * Storey [2002] J. D. Storey.  A direct approach to false discovery rates.  _Journal of the Royal Statistical Society: Series B (Statistical Methodology)_ , 64(3):479–498, 2002.
  * Tian and Ramdas [2019] J. Tian and A. Ramdas.  ADDIS: an adaptive discarding algorithm for online FDR control with conservative nulls.  _Advances in Neural Information Processing Systems_ , 32, 2019.
  * Tian and Ramdas [2021] J. Tian and A. Ramdas.  Online control of the familywise error rate.  _Statistical Methods in Medical Research_ , 30(4):976–993, 2021.
  * Tukey [1953] J. W. Tukey.  _The collected works of John W. Tukey. Volume VIII. Multiple comparisons: 1948–1983_.  Chapman and Hall, 1953.
  * Weinstein and Ramdas [2020] A. Weinstein and A. Ramdas.  Online control of the false coverage rate and false sign rate.  In _International Conference on Machine Learning_ , pages 10193–10202. PMLR, 2020.
  * Wellcome Trust Case Control Consortium et al. [2007] Wellcome Trust Case Control Consortium et al.  Genome-wide association study of 14,000 cases of seven common diseases and 3,000 shared controls.  _Nature_ , 447:661–678, 2007.
  * Xu and Ramdas [2022] Z. Xu and A. Ramdas.  Dynamic algorithms for online multiple testing.  In _Mathematical and Scientific Machine Learning_ , pages 955–986. PMLR, 2022.
  * Yang et al. [2017] F. Yang, A. Ramdas, K. Jamieson, and M. Wainwright.  A framework for Multi-A(rmed)/B(anit) testing with online FDR control.  In _Advances in Neural Information Processing Systems 30_ , pages 5959––5968, 2017.  URL https://papers.nips.cc/paper/7177-a-framework-for-multi-armedbandit-testing-with-online-fdr-control.pdf.
  * Zehetmayer et al. [2021] S. Zehetmayer, M. Posch, and F. Koenig.  Online control of the false discovery rate in group-sequential platform trials.  _arXiv preprint arXiv:2112.10619_ , 2021.
  * Zhao et al. [2019] Q. Zhao, D. S. Small, and W. Su.  Multiple testing when many p-values are uniformly conservative, with application to testing qualitative interaction in educational interventions.  _Journal of the American Statistical Association_ , 114(527):1291–1304, 2019.
  * Zrnic et al. [2020] T. Zrnic, D. Jiang, A. Ramdas, and M. Jordan.  The power of batching in multiple hypothesis testing.  In _International Conference on Artificial Intelligence and Statistics_ , pages 3806–3815. PMLR, 2020.
  * Zrnic et al. [2021] T. Zrnic, A. Ramdas, and M. I. Jordan.  Asynchronous online testing of multiple hypotheses.  _JMLR_ , 22:33–1, 2021.

## Appendix

Figure 5: FDR of LORD, SAFFRON and ADDIS compared with uncorrected testing,
the BH procedure and alpha-spending as the proportion of non-nulls $\pi_{1}$
varies. The solid red horizontal line gives the target level of $\alpha=0.05$.
We set $T=1000$ and results are based on $10^{4}$ simulation replicates.