# Balancing central and marginal rejection when combining independent
significance tests

Chris Salahub  Department of Statistics and Actuarial Science, University of
Waterloo R. Wayne Oldford  Department of Statistics and Actuarial Science,
University of Waterloo

###### Abstract

A common approach to evaluating the significance of a collection of $p$-values
combines them with a pooling function, in particular when the original data
are not available. These pooled $p$-values convert a sample of $p$-values into
a single number which behaves like a univariate $p$-value. To clarify
discussion of these functions, a telescoping series of alternative hypotheses
are introduced that communicate the strength and prevalence of non-null
evidence in the $p$-values before general pooling formulae are discussed. A
pattern noticed in the UMP pooled $p$-value for a particular alternative
motivates the definition and discussion of central and marginal rejection
levels at $\alpha$. It is proven that central rejection is always greater than
or equal to marginal rejection, motivating a quotient to measure the balance
between the two for pooled $p$-values. A combining function based on the
$\chi^{2}_{\kappa}$ quantile transformation is proposed to control this
quotient and shown to be robust to mis-specified parameters relative to the
UMP. Different powers for different parameter settings motivate a map of
plausible alternatives based on where this pooled $p$-value is minimized.

##  1 Introduction

When presented with a collection of $p$-values, a natural question is whether
they constitute evidence as a whole against the null hypothesis that there are
no significant results. The multiple testing problem arises because answering
this question requires different analysis than univariate $p$-values. A
univariate threshold applied to all $p$-values, for example, will no longer
control the type I error at the level of the threshold. A common approach to
control the type I error (often called the family-wise error rate in this
context) is to use a function to combine the $p$-values into a single value
which behaves like a univariate $p$-value.

Explicitly, consider a collection of $M$ independent test statistics
$\mathbf{t}={(t_{1},\dots,t_{M})}^{\mkern-1.5mu\mathsf{T}}$ having $p$-values
$\mathbf{p}={(p_{1},\dots,p_{M})}^{\mkern-1.5mu\mathsf{T}}$ for the null
hypotheses $H_{01}$, $H_{02}$, $\dots$, $H_{0M}$ – for example, $\chi^{2}$
tests for the association of $M$ individual genes with the presence of a
disease where each $H_{0i}$ asserts no association. Assessing the overall
significance of $\mathbf{p}$ while controlling the family-wise error rate
(FWER) at the outset of analysis is common practice in meta-analysis and big
data applications (Heard and Rubin-Delanchy, 2018; Wilson, 2019). The FWER is
the probability of rejecting one or more of $H_{01},\dots,H_{0M}$ when all are
true, equivalent to the type I error of the joint hypothesis

$H_{0}=\cap_{i=1}^{M}H_{0i}.$

To emphasize the null distributions, $p_{i}\sim U=Unif(0,1)$ for all
$i\in\\{1,\dots,M\\}$, this is often written

$H_{0}:p_{1},p_{2},\dots,p_{M}\overset{\mathrm{iid}}{\sim}U.$

To test $H_{0}$, a statistic $l(\mathbf{p}):[0,1]^{M}\mapsto\mathbb{R}$ of the
$p$-values with a distribution that is known or easily simulated under $H_{0}$
can be computed. If $l(\mathbf{p})$ has cumulative distribution function (CDF)
$F_{l}(l)$ under $H_{0}$, then $l(\mathbf{p})$ admits
$g(\mathbf{p})=1-F_{l}(l(\mathbf{p}))\sim Unif(0,1)$ such that rejecting
$H_{0}$ when $g(\mathbf{p})\leq\alpha$ controls the FWER at level
$\alpha$.111Note that the use of the CDF in
$g(\mathbf{p})=1-F_{l}(l(\mathbf{p}))$ implies that $g(\mathbf{p})$ is
identical for any statistic that is a monotonic transformation of
$l(\mathbf{p})$. $g(\mathbf{p})$ therefore summarizes the evidence against
$H_{0}$ in a statistic which behaves like a univariate $p$-value: its
magnitude is inversely related to its significance.

If we want $g(\mathbf{p})$ to additionally have convex acceptance regions like
a univariate $p$-values, it should be continuous in each argument and
monotonically non-decreasing, i.e. $g(p_{1},\dots,p_{M})\leq
g(p_{1}^{*},\dots,p_{M}^{*})\leftrightarrow p_{1}\leq
p_{1}^{*},\dots,p_{M}\leq p_{M}^{*}$. Functions failing these can behave
counter-intuitively, as they may accept $H_{0}$ for small $p_{i}$ only to
reject as $p_{i}$ increases for some margin $i$. Finally, if there is no
reason to favour any margin, $g$ should be symmetric in $\mathbf{p}$. The term
evidential statistic refers to $g(\mathbf{p})$ meeting these criteria
generally (Goutis et al., 1996), and when testing $H_{0}$ they are called
pooled $p$-values. There is no lack of pooled $p$-value proposals, including
the statistics of Tippett (1931), Fisher (1932), Pearson (1933), Stouffer et
al. (1949), Mudholkar and George (1977), Heard and Rubin-Delanchy (2018), and
Cinar and Viechtbauer (2022).

As all of these methods have convex acceptance regions and control the FWER at
$\alpha$ under the rule $g(\mathbf{p})\leq\alpha$, statistical power against
alternative hypotheses is often used to distinguish them. Ideally, one among
them would be uniformly most powerful (UMP) against a very broad alternative
but this is not possible because of the generality of $H_{0}$. Indeed,
Birnbaum (1954) proves that if all $f_{i}$ are strictly non-increasing so that
$p_{i}\sim f_{i}$ is biased to small values when $H_{0i}$ is false, then there
is no UMP test against the negation of $H_{0}$,

$H_{1}=\neg H_{0}:p_{1}\sim f_{1},p_{2}\sim f_{2},\dots,p_{M}\sim f_{M}$

where $f_{i}\neq U$ for at least one $i\in\\{1,\dots,M\\}$. As the simulation
studies in Westberg (1985), Loughin (2004), and Kocak (2017) readily
demonstrate, the number of false $H_{0i}$ and the non-null distributions
$f_{i}$ together specify the unique most powerful test. For the particular
case of testing $H_{0}$ against $H_{1}$ with
$f_{1}=f_{2}=\dots=f_{M}=Beta(a,b)$ for $a\in(0,1]$ and $b\in[1,\infty)$, the
Neyman-Pearson lemma proves that the pooled $p$-value
$HR\left(\mathbf{p};w\right)$ induced by the statistic

$l_{HR}(\mathbf{p};w)=w\sum_{i=1}^{M}\ln
p_{i}-(1-w)\sum_{i=1}^{M}\ln(1-p_{i})$ (1)

with $w=(1-a)/(b-a)\in[0,1]$ is uniformly most powerful (UMP) (Heard and
Rubin-Delanchy, 2018).

Though $HR\left(\mathbf{p};(1-a)/(b-a)\right)$ is UMP against $H_{1}$ for
$f_{1}=\dots=f_{M}=Beta(a,b)$, it is rarely assumed that $f_{1}=\dots=f_{M}$
in the pairwise search for variables. Rather, some of these are assumed to be
non-uniform while others are assumed null. A discussion of this setting
therefore requires measures of the prevalence and strength of evidence against
$H_{0}$, captured by a series of telescoping alternative hypotheses that
bridge the gap between $H_{1}$ and the setting where
$HR\left(\mathbf{p};w\right)$ is UMP.

Section 2 introduces the measures of strength and prevalence used in this
paper, as these are required to understand central and marginal rejection
later. Prevalence is measured by $\eta$, the proportion of non-null tests,
while strength is measured by the Kullback-Leibler divergence. Assuming that
non-null tests come from a restricted beta family, the power of a UMP method
for particular beta distributions is investigated for different values of the
prevalence and strength in Section 4. A pattern of high power for either
strong evidence in a few tests or weak evidence in many tests is noticed and
developed into a framework for choosing pooled $p$-values in Section 5.
Following the necessary definitions to develop this framework, including the
concepts of central and marginal rejection, it is proved that the tendency to
reject concentrated evidence is always less than diffuse evidence, allowing
for the definition of a coefficient of the preference of a pooling function to
diffuse evidence.

Section 6 proposes a pooling function based on the $\chi^{2}$ quantile
transformation which controls this preference through its degrees of freedom.
It is proven that large degrees of freedom give a pooling function which
prefers diffuse evidence while small degrees of freedom prefer concentrated
evidence. In a simulation study, this proposal is shown to nearly match the
UMP when correctly specified and is more robust to errors in specification.
These conclusions are extended in Section 7 where a sweep of parameter values
is used to identify the the most powerful choice for a given sample and
suggest a region of most plausible alternative hypotheses within the framework
in light of it.

##  2 Measuring the strength and prevalence of evidence

When proving that no UMP exists for the general hypothesis $H_{1}=\neg H_{0}$,
Birnbaum (1954) provides a couple of two-dimensional examples. Though these
are demonstrative, they are not instructive for the discussion of tests
generally. To the same conclusions, each of Westberg (1985), Loughin (2004),
and Kocak (2017) simulate a variety of populations with differing proportions
of $\mathbf{p}$ generated under $U$ or some alternative distribution. We begin
by defining a telescoping series of alternative hypotheses which capture the
settings explored in these empirical investigations.

###  2.1 Telescoping alternatives

Starting at $H_{1}$, assume $H_{0i}$ is false only for $i\in
J\subset\\{1,\dots,M\\}$ and quantify the proportion of non-null hypotheses by
$\eta=\lvert{J}\rvert/M$. This implies an alternative hypothesis

$H_{2}:p_{i}\sim\begin{cases}f_{i}\neq U&\text{ if }i\in J,\\\ U&\text{ if
}i\notin J.\end{cases}$

As no distinctions between $H_{01},\dots,H_{0M}$ are made in $H_{0}$, $\eta$
captures the prevalence of evidence against $H_{0}$ without loss of
generality. If it is additionally assumed that all $i\in J$ have the same
alternative distribution $f\neq U$, this gives the alternative hypothesis

$H_{3}:p_{i}\sim\begin{cases}f&\text{ if }i\in J,\\\ U&\text{ if }i\notin
J.\end{cases}$

Finally, in the particular case where $\eta=1$, $\lvert{J}\rvert=M$ and

$H_{4}:p_{1},p_{2},\dots,p_{M}\overset{\mathrm{iid}}{\sim}f\neq U$

is obtained. Though restricted compared to $H_{1}$, this alternative makes
sense for meta-analysis or repeated experiments, where we could assume all
$p_{i}$ are independently and identically distributed when $H_{0}$ is false.

$H_{4}$ was distinguished from $H_{1}$ as early as Birnbaum (1954) (there
called $H_{A}$ and $H_{B}$ respectively), but no exploration of intermediate
possibilities was considered. All of Westberg (1985), Loughin (2004), and
Kocak (2017) explore factorial combinations of $\eta$ and $f$, and so use
instances of $H_{3}$ for their investigations. When testing $H_{4}$ against
$H_{0}$, Heard and Rubin-Delanchy (2018) prove $HR\left(\mathbf{p};w\right)$
is the UMP pooled $p$-value if $f$ is from a constrained beta family. By
stating clearly $H_{1}\supset H_{2}\supset H_{3}\supset H_{4}$, a framework
for alternative hypothesis is created that contextualizes and relates these
previous results. Additionally, the parameter $\eta$ under $H_{3}$ naturally
measures the prevalence in $\mathbf{p}$ of evidence against $H_{0}$.

###  2.2 Measuring the strength of evidence

Intuitively, if $f$ has a density highly concentrated near zero then it
provides “strong” evidence against $H_{0}$. This is because $p$-values
generated by $f$ will tend to be smaller than those following $U$ and
therefore will be rejected more frequently for any $\alpha$. Any measure of
the strength of evidence in $f$ should therefore increase as the magnitude of
$f$ for small values increases.

This relatively simple criterion is challenging to apply to $H_{2}$. Every
$f_{i}$ for $i\in J$ may be distinct and a single value characterizing their
multiple, potentially very different, departures from $U$ introduces
ambiguity. Taking a mean of measures, for example, conflates different
instances of $H_{2}$. If $J=\\{1,2\\}$, the mean strength of evidence cannot
distinguish strong evidence in $f_{1}$ with weak evidence in $f_{2}$ from
moderate evidence in both. This difficulty is avoided if all $f_{i}$ are
equal, i.e. if $H_{3}$ is chosen as the alternative hypothesis. Indeed, this
choice is common in previous empirical investigations.

Westberg (1985) generates $\mathbf{p}$ by testing the difference in means of
two simulated normal samples, and measures the strength of evidence by the
true difference in means between the generative distributions. This is
reasonable, but limits us to tests comparing population parameters and
requires assumptions on $\mathbf{t}$ (the tests generating $\mathbf{p}$).
Considering $f$ directly, Loughin (2004) takes
$p_{i}\overset{\mathrm{iid}}{\sim}Beta(a,b)$ for all $i\in J$, restricts
$a=1\leq b$ so that $f$ is non-increasing, and measures the strength of
evidence with one minus the median of $f$: $1-0.5^{1/b}$.222Kocak (2017) also
uses $p_{i}\sim Beta(a,b)$ but takes the broader $0<a\leq b$ and does not
attempt to measure the strength of $f$’s departure from $U$. This measure of
strength is limited to $Beta(1,b)$, though it does achieve the intuitive
ordering desired. A more general measure that applies to any $f$ is the
Kullback-Leibler (KL) divergence, given by

$D(p,q)=\int_{\mathcal{X}}p(x)\ln\left(\frac{p(x)}{q(x)}\right)dx$

for $q(x)$ to density $p(x)$ with mutual support on $\mathcal{X}$.

Widespread application of the KL divergence in information theory and machine
learning aside, one interpretation of this measure suits pooled hypothesis
testing nicely. Joyce (2011) describes the KL divergence as the extra
information encoded in $q(x)$ when expecting $p(x)$. The explicit assumption
underlying the pooled test of $H_{0}$ is that $f_{i}=U$ for all
$i\in\\{1,\dots,M\\}$, which gives a natural expected density $q(x)=U(x)$.
Furthermore, this density is, in some sense, minimally informative: no region
of $[0,1]$ is distinguished from any other by $U$. Any additional information
which discriminates particular regions of $[0,1]$, in particular values near
$0$, will help inform rejection.

###  2.3 Choosing a family for the alternative distribution

The beta family of distributions is appealing as a model for the alternative
distribution $f$ under $H_{3}$ for two main reasons. First, it has the same
support as $U$ without the need for adjustment. Second, a wide variety of
different density shapes can be achieved by changing its two parameters and it
has a non-increasing density whenever $a\leq 1\leq b$. This latter quality
makes it ideal to model alternative $p$-value distributions under the
assumption that $p_{i}$ is biased to small values when $H_{0}$ is false. These
features are likely why it is commonly used in the literature (e.g. Loughin
(2004), Kocak (2017)). More significantly, the Neyman-Pearson lemma proves
that $HR\left(\mathbf{p};w\right)$ is UMP for
$p_{1},\dots,p_{M}\overset{\mathrm{iid}}{\sim}Beta(a,1/w+a(1-1/w))$ when
$0\leq w,a\leq 1$, and so a best-case benchmark for power exists to compare to
other tests (Heard and Rubin-Delanchy, 2018). Were another distribution
chosen, this important reference point would be absent.

When $f=Beta(a,b)$ the KL divergence also has a relatively simple expression.
Letting $u(x)=1$ be the uniform density and
$f(x)=\frac{\Gamma(a+b)}{\Gamma(a)\Gamma(b)}x^{a-1}(1-x)^{b-1}$ be the beta
density with parameters $a$ and $b$, $D(u,f)$ is given by

$D(u,f)=-\int_{0}^{1}\ln
f(x)dx=a+b+\ln\left(\frac{\Gamma(a)\Gamma(b)}{\Gamma(a+b)}\right)-2.$

For the case of the beta densities where $HR\left(\mathbf{p};w\right)$ is UMP,
i.e. $a\leq 1\leq b$, we can express this in terms of $a$ and $w=(1-a)/(b-a)$:

$D(u,f):=D(a,w)=2a+\frac{1-a}{w}+\ln\left(\frac{\Gamma(a)\Gamma\left(\frac{1}{w}+a\left[1-\frac{1}{w}\right]\right)}{\Gamma\left(2a+\frac{1-a}{w}\right)}\right)-2.$
(2)

This is a less convenient expression, but provides a direct link between the
strength of evidence $D(a,w)$ and the UMP test against $H_{4}$ by the shared
parameter $w$. Interestingly, though the strength depends on both $a$ and $w$,
the UMP test only depends on the latter.

To visualize the strength of evidence provided by different beta
distributions, shaded inset densities for different choices of $w$ and the KL
divergence $D(a,w)$ are displayed in Figure 1 placed at $\ln(w)$, $\ln
D(a,w)$.333The densities of these inset plots were determined for each $w$,
$D(a,w)$ pair by finding the corresponding $a$ value numerically using
Equation (2). This is the cause of the irregular plots in the upper right
corner: when $w$ is large enough the required $a$ to obtain a set $D(a,w)$ is
too small to be represented as a floating point double alongside $w\approx 1$.
This is inconvenient, but these cases correspond to densities that are
effectively degenerate at zero in any case. When $a=w=1$, $f=u$ so $D(a,w)=0$.
Decreasing either of $a$ or $w$ from 1 causes a larger magnitude of $f$ near
zero, with the limiting case $a=w=0$ corresponding to a degenerate
distribution at zero. This general trend in shapes is seen in Figure 1,
decreasing $w$ or increasing $D(a,w)$ increases the concentration of $f$ near
zero.

Figure 1: Densities and log KL divergences of $Beta(a,1/w+a(1-1/w))$ from $U$
by $w$. Insets display the densities over $[0,1]$ horizontally and $[0,2]$
vertically and are centred at the $\ln(w)$, $\ln D(a,w)$ coordinates
corresponding to the density. These densities range from nearly vertical at 0
when $D(a,w)\approx e^{5}$ to nearly uniform when $D(a,w)\approx e^{-5}$.

This suggests a limitation in the KL divergence. Though the ordering of beta
densities by $D(a,w)$ generally conforms to the intuitive rule – larger
divergences correspond to inset densities with greater magnitude near zero –
the parameter $w$ is still relevant to the shape. The KL divergence does not
distinguish between departures from uniform near 1 and near 0, despite their
relevance for rejection when, for example, rejecting the null hypotheses of
$p$-values below a threshold. This is particularly obvious in the final row of
inset plots in Figure 1. When $\ln(w)=-6$, the density for $\ln D(a,w)=-5$ is
mostly flat, with a slight increase in density near zero and a large decrease
near one. When $\ln(w)=0$, however, a much larger spike in the density near
zero is present.

Nonetheless, the ordering on beta densities imposed by the KL divergence is
still very informative. It classifies, generally, which densities are biased
to small values. Therefore, $D(a,w)$ provides a convenient measure of the
strength of evidence contained in $f$ under the alternative hypothesis, and
has computationally convenient form for the case of interest where
$f=Beta(a,b)$.

##  3 Pooled $p$-values

Having explored the possible alternatives to $H_{0}$ and some specific
instances of these alternatives, we can now focus on the pooled $p$-values
meant to test $H_{0}$ against these alternatives. Recall that any pooled
$p$-value $g(\mathbf{p})$ is derived from the null distribution of a
corresponding statistic $l(\mathbf{p})$. The statistics underlying pooled
$p$-values are of two basic kinds, based either on the $k^{th}$ order
statistic $p_{(k)}$ of $\mathbf{p}$ (e.g. Tippett (1931) and Wilkinson (1951))
or on transformations of each $p_{i}$ using some quantile function $F^{-1}(p)$
(e.g. Fisher (1932), Pearson (1933), Stouffer et al. (1949), Lancaster (1961),
Edgington (1972), Mudholkar and George (1977), Heard and Rubin-Delanchy
(2018), Wilson (2019), and Cinar and Viechtbauer (2022)). The former case
takes the general form

$ord\left(\mathbf{p};k\right)=\sum_{l=k}^{M}{M\choose
l}p_{(k)}^{l}(1-p_{(k)})^{M-l},$ (3)

and the latter

$g(\mathbf{p})=1-F_{M}\left(\sum_{i=1}^{M}c_{i}F^{-1}(1-p_{i})\right)$ (4)

where $c_{1},\dots,c_{M}\in\mathbb{R}$ are known constants, typically
$c_{1}=\dots c_{M}=1$. Equation (3) gives the pooled $p$-value based on
$l_{Ord}(\mathbf{p};k)=p_{(k)}$, as with
$Tip(\mathbf{p})=ord\left(\mathbf{p};1\right)=1-(1-p_{(1)})^{M}$ (Tippett,
1931), while different choices of $F$ and $F_{M}$ in Equation (4) give the
pooled $p$-value based on $l(\mathbf{p})=\sum_{i=1}^{M}F^{-1}(1-p_{i})$.

Obvious choices include the normal and gamma families, as these are closed
under addition. For example, letting $\Phi$ be the $N(0,1)$ CDF and choosing
$F(x)=\Phi(x)$ and $F_{M}(x)=\Phi(x/\sqrt{M})$ gives

$Sto(\mathbf{p})=1-\Phi\left(\sum_{i=1}^{M}\Phi^{-1}(1-p_{i})/\sqrt{M}\right)$
(5)

based on $l_{Sto}(\mathbf{p})=\sum_{i=1}^{M}\Phi(1-p_{i})$ from Stouffer et
al. (1949). Letting $G_{k,\theta}(x)$ be the CDF of the gamma distribution
with shape parameter $k$ and scale parameter $\theta$, taking
$F(x)=G_{k,\theta}(x)$ and $F_{M}(x)=G_{Mk,\theta}(x)$ gives

$gam(\mathbf{p})=1-G_{Mk,\theta}\left(\sum_{i=1}^{M}G_{k,\theta}^{-1}(1-p_{i})\right)$
(6)

based on $l_{gam}(\mathbf{p})=\sum_{i=1}^{M}G_{k,\theta}^{-1}(1-p_{i})$. $gam$
requires the choice of a parameter $\theta$; choosing $\theta=1$ gives the
gamma method from Zaykin et al. (2007) while $k=1$ and $\theta=2$ gives
Fisher’s method from Fisher (1932).444Should the $p$-values be weighted for
some reason, the gamma distribution also allows more stable weighting than the
constants $c_{1},\dots,c_{M}$ in Equation (4) by analogously giving each
$p_{i}$ an individual shape parameter $k_{i}$ (or equivalently $\chi^{2}$
degrees of freedom $\kappa_{i}$) (Lancaster, 1961).

R. A. Fisher’s method deserves some additional consideration alongside an
analogous proposal from Karl Pearson around the same time555Owen (2009) notes
that Pearson’s proposal is actually slightly different than has been credited
in the literature following Birnbaum (1954). Though the use of
$\sum_{i=1}^{M}\ln(p_{i})$ was suggested by Karl Pearson, it was in the
context of comparing the value to $\sum_{i=1}^{M}\ln(1-p_{i})$ and taking the
minimum of the two. Owen (2009) develops this idea into a series of pooling
functions that perform best for concordant or discordant effect estimates in a
regression setting.. Both of $l_{Fis}(\mathbf{p})=-2\sum_{i=1}^{M}\ln p_{i}$
(Fisher, 1932) and $l_{Pea}(\mathbf{p})=-2\sum_{i=1}^{M}\ln(1-p_{i})$
(Pearson, 1933) were originally proposed only as computational tricks for the
distribution of $\prod_{i=1}^{M}p_{i}$ (Wallis, 1942), but are also quantile
transformations based on the $\chi_{2}^{2}$ distribution. Let

$F_{\chi}(x;\kappa)=\int_{0}^{x}\frac{1}{2^{\kappa/2}\Gamma(\kappa/2)}t^{\kappa/2-1}e^{-t/2}dt,$
(7)

be the CDF of the $\chi^{2}_{\kappa}$ distribution, in particular
$F_{\chi}(x;2)=1-e^{-x/2}$. Therefore, $F^{-1}_{\chi}(1-p;2)=-2\ln p$ and so
taking $F(x)=F_{\chi}(x;2)$ and $F_{M}(x)=F_{\chi}(x;2M)$ gives

$Fis(\mathbf{p})=1-F_{\chi}\big{(}l_{F}(\mathbf{p});2M\big{)}=1-F_{\chi}\left(-2\sum_{i=1}^{M}\ln
p_{i};2M\right)=1-F_{\chi}\left(\sum_{i=1}^{M}F_{\chi}^{-1}(1-p_{i};2);2M\right)$

consistent with Equation (4). In contrast, $l_{Pea}(\mathbf{p})$ uses lower
tail probabilities by taking $F_{\chi}(p_{i};2)$, and so

$Pea(\mathbf{p})=F_{\chi}\left(\sum_{i=1}^{M}F_{\chi}^{-1}(p_{i};2);2M\right)$

departs from the general quantile transformation equation.

##  4 Benchmarking the most powerful test

Alone, $Fis(\mathbf{p})$ is preferred to $Pea(\mathbf{p})$, as Birnbaum (1954)
found $Pea(\mathbf{p})$ inadmissible for the alternative hypothesis $H_{1}$ if
the $t_{i}$ independently follow particular distributions in the exponential
family. $Fis(\mathbf{p})$, in contrast, was admissible in this setting and is
optimal in some sense for others (Littell and Folks, 1971; Koziol and Perlman,
1978). Together, the statistics for these two pooled $p$-values are combined
in
$l_{HR}(\mathbf{p};w)=-\frac{w}{2}l_{Fis}(\mathbf{p})+\frac{1-w}{2}l_{Pea}(\mathbf{p})$,
the UMP statistic under $H_{4}$ when $f=Beta(a,1/w+a(1-1/w))$ (Heard and
Rubin-Delanchy, 2018). Intuitively, then, $HR\left(\mathbf{p};w\right)$ is a
test based on a linear combination of the lower and upper tail probabilities
of $\mathbf{p}$ transformed to $\chi^{2}_{2}$ quantiles. When $w=1$ it
considers the upper tail alone and when $w=0$ the lower tail alone. Note that
$l_{HR}(\mathbf{p};w)$, the statistic, and $HR\left(\mathbf{p};w\right)$, the
unique corresponding pooled $p$-value, will be used interchangeably thoughout
this paper.

Unfortunately, this imbues $HR\left(\mathbf{p};w\right)$ with some practical
shortcomings. While both $l_{Fis}(\mathbf{p})$ and $l_{Pea}(\mathbf{p})$ are
$\chi^{2}_{2M}$ distributed under $H_{0}$, they are not independent and so
their combination in $l_{HR}(\mathbf{p};w)$ does not have a closed-form
distribution. Approximation as in Mudholkar and George (1977) or simulation
must be used to determine the $\alpha$ quantiles or visualize their
distribution. This means, for example, the kernel density estimates of
$l_{HR}(\mathbf{p};w)$ by $w$ in Figure 2 required the generation of 100,000
independent simulated samples of the case
$p_{1},\dots,p_{10}\overset{\mathrm{iid}}{\sim}U$. Further, $H_{4}$ is the
least general of the telescoping alternative hypotheses $H_{1}\supset
H_{2}\supset H_{3}\supset H_{4}$ and is a less natural choice than $H_{3}$ if
only a subset of tests are thought to be significant. Empirical and
theoretical investigations show the most powerful test depends on $\eta$ and
$f$ under $H_{3}$, so $HR\left(\mathbf{p};w\right)$ may be less exceptional
under this more general hypothesis. Finally, $HR\left(\mathbf{p};w\right)$ is
only UMP if $w$ is known, which is seldom true in practice.

Figure 2: Densities of $l_{HR}$ by $w$ when $M=10$. Solid lines indicate
$w=e^{-6}$, dashed lines $w=e^{-3}$, dotted lines $w=1/2$, and dot-dashed
lines $w=1$. Note how $w=1$ and $w=e^{-6}$ are nearly mirrored distributions
skewed away from zero and $w=1/2$ is symmetric at zero.

These practical difficulties manifest in two possible errors, assuming $H_{4}$
when $H_{3}$ is true and choosing $w$ when the true parameter is $\omega$, and
four cases of mis-specification depending on which is present. The power of
$HR\left(\mathbf{p};w\right)$ under all four cases was investigated by a
simulation study at level $\alpha=0.05$. For both of $H_{3}$ and $H_{4}$ and a
range of mis-specified $w$, $HR\left(\mathbf{p};w\right)$ was applied to
factorial combinations of $D(a,w)$, $w$, and $M$ covering their respective
ranges. $D(a,w)$ was chosen on the log scale ranging from $-5$ to 5 at 0.5
increments, $w$ was chosen on the log scale at values $-6,-5,\dots,0$, and the
values of $M$ were 2, 5, 10, and 20.

For each of the parameter settings, $l_{HR}(\mathbf{p};w)$’s 0.95 quantiles
under $H_{0}$ given $w$ and $M$ are simulated by generating 100,000
independent samples
$\mathbf{p}_{i}=p_{i1},\dots,p_{iM}\overset{\mathrm{iid}}{\sim}U$, computing
$l_{HR}(\mathbf{p}_{i};w)=l_{HRi}$, and taking the 0.95 quantile of the
sequence $l_{HR1},\dots,l_{HR100,000}$ as the 0.95 quantile of
$l_{HR}(\mathbf{p};w)$ under $H_{0}$. Note that the value of $a$ and the case
do not impact this simulation, and so these quantiles are used across all $a$
values under both $H_{3}$ and $H_{4}$.

Next a Monte Carlo estimate of the probability of rejecting $H_{0}$ using
$l_{HR}(\mathbf{p};w)$ (i.e. the power of $l_{HR}(\mathbf{p};w)$) is generated
for each case. The details of this estimate depend on whether $H_{3}$ or
$H_{4}$ was used to generate the data and whether $w$ was known or not when
choosing $l_{w}$. To begin, consider the benchmark case when $w$ is known and
the data are generated according to $H_{4}$.

###  4.1 Case 1: correct hypothesis and $w$

If the data are generated under $H_{4}$ and $w$ is chosen correctly,
$HR\left(\mathbf{p};w\right)$ is UMP and so provides the greatest power of any
test. In this case, the probability of rejection for a given $a,w$, and $M$
setting is estimated by generating 10,000 independent samples
$\mathbf{p}_{i}=p_{i1},\dots,p_{iM}\overset{\mathrm{iid}}{\sim}Beta(a,1/w+a(1-1/w))$.
$l_{HR}(\mathbf{p}_{i};w)$ is computed for each sample and compared to the
simulated 0.95 quantile of $l_{HR}(\mathbf{p};w)$ under $H_{0}$666This
corresponds to testing whether $HR\left(\mathbf{p}_{i};w\right)\leq 0.05$.. If
the value is larger than the quantile $H_{0}$ is correctly rejected and if it
is smaller the test incorrectly fails to reject $H_{0}$. Power is estimated as
the proportion of the 10,000 generated samples which correctly lead to the
rejection of $H_{0}$, giving a worst-case standard error less than 0.005 based
on the binomial distribution.

|
---|---
(a) | (b)
Figure 3: Power of $HR\left(\mathbf{p};w\right)$ by the KL divergence coloured
by $w$ and scaled by $M$ displayed using (a) power curves when $M=2$ joined by
$w$ and (b) lines from $M=2$ to $M=20$ for $w=e^{-6}$ and $0$. Increasing
either $M$ or the KL divergence increases the power and the greatest rate of
change in both occurs when the divergence is in the interval $(e^{-2},e^{2})$.

This procedure is applied to all settings of $M$, $w$, and $D(a,w)$ outlined
in Section 4, corresponding to the beta densities of Figure 1. The beta
parameter $a$ was determined for a given $w$ and $D(a,w)=D$ by finding the
root of $f(x)=D(x,w)-D$, while the parameter $b$ is given by $1/w+a(1-1/w)$.
This results in an imbalance in the settings, as $w>e^{-4}$ require $a$ less
than the typical floating point minimum value to achieve $D(a,w)=5$. The
impact on the coverage of $D(a,w)$ for each $w$ choice as a result of this,
however, was slight as shown by the small gap in plots in Figure 1.

Figure 3(b) shows a scatterplot of the power of $HR\left(\mathbf{p};w\right)$
by $D(a,w)$ for every setting when $M=2$ and $M=20$, and Figure 3(a) shows the
power curves of $HR\left(\mathbf{p};w\right)$ by $D(a,w)$ coloured by $w$.
Generally, power increases in both $M$ (the number of $p$-values) and $D(a,w)$
(the KL divergence), which is unsurprising. Decreasing $D(a,w)$ for
$f=Beta(a,1/w+a(1-1/w))$ necessarily gives a density closer to $u$ which is
therefore less likely to cause rejection, thus reducing the power. At a
certain threshold on $D(a,w)$, $f\approx u$ and so the power will be
$\sim\alpha$ for all $D(a,w)$ less than the threshold. Similarly, when
$D(a,w)$ is large enough, rejection occurs almost certainly and so the power
is constant at one. This suggests that most changes in the power of
$HR\left(\mathbf{p};w\right)$ occur for moderate levels of evidence; when the
evidence is too weak or too strong, all pooled $p$-values will perform equally
poorly or well. Regardless of $D(a,w)$, increasing the number of $p$-values
makes any distributional differences between $f$ and $u$ more easily
detectable, as the whole sample KL divergence is given by $MD(a,w)$. This is
why the impact of $M$ on the power in Figure 3(b) increases in $D(a,w)$.

An interesting feature of Figure 3(a) is the ordering of the lines by
decreasing $w$ for essentially every KL divergence. This pattern holds almost
everywhere with the exception of several crossings of the lowest power curves.
Referring to Figure 1, increasing $w$ for a given $D(a,w)$ increases the
magnitude of the density near zero, thereby increasing the probability of
extremely small $p$-values. This difference was noted in Section 2.2, and
causes the expected increase in the power of the UMP.

###  4.2 Case 2: correct hypothesis with mis-specified $w$

Of course, the curves of Figure 3(a) are not realistic. In practice, $D(a,w)$
is set by the data generating process and we lack the perfect knowledge of $w$
and $f$ needed to attain them. Suppose that $\mathbf{p}$ is generated under
$H_{4}$ with $f=Beta(a,\beta)$ and $a\in[0,1]$, $\beta\in[1,\infty)$ but that
$HR\left(\mathbf{p};w\right)$ is used instead of the correct
$HR\left(\mathbf{p};\omega\right)$ with $\omega=(1-a)/(\beta-a)$. Though
$HR\left(\mathbf{p};w\right)$ is from the same family of tests, it is no
longer UMP and so will not match the power achieved by
$HR\left(\mathbf{p};\omega\right)$.

The reduction of power from using $HR\left(\mathbf{p};w\right)$ when the UMP
is $HR\left(\mathbf{p};\omega\right)$ for each $a$, $w$, $\omega$, and $M$
setting from Section 4.1 was determined by generating 10,000 independent
samples
$p_{i1},\dots,p_{iM}\overset{\mathrm{iid}}{\sim}Beta(a,1/\omega+a(1-1/\omega))$
and computing $HR\left(\mathbf{p}_{i};w\right)$. The proportion of samples
rejected based on the simulated null 0.95 quantiles was recorded as the power
of $HR\left(\mathbf{p};w\right)$ when data were truly generated with the
parameter $\omega$. This procedure was repeated for each of
$w=e^{-6},e^{-3},1/2,$ and $1$ for every parameter setting. Figure 4 displays
the results when $M=2$ for $\omega=e^{-6}$ and $1$. As one setting of $w$
matches $\omega$ in this case, the highest curve displays the power of the
UMP.

Figure 4: Power curves for $HR\left(\mathbf{p};w\right)$ against $D(a,\omega)$
when $M=2$ with colours giving the value of $w$ and points along the curves
giving $\omega$. Mis-specification of $\omega$ has less impact on power than
the non-null distribution $f$, but the greater the difference between $w$ and
$\omega$, the greater the reduction in power.

Two patterns stand out in this plot. First, it is clear that mis-specification
impacts the power less than the distribution of $p$-values under $H_{4}$.
Despite the incorrect value $w$ in $HR\left(;w\right)$, the mis-specified
curves have the same shape in $D(a,\omega)$ as the UMP
$HR\left(;\omega\right)$. In most cases, mis-specification results in only a
slight decrease in power.

Second, larger mis-specifications lead to larger decreases in power. The
curves for every $w$ are ordered by their distance from $\omega$ for both
$\omega=1$ and $e^{-6}$. When $\omega=1$, for example, the lines are ordered
so $w=1$ has the greatest power followed closely by $w=1/2$ and more distantly
by $e^{-3}$ and $e^{-6}$. This pattern is reversed when $\omega=e^{-6}$. In
both cases, mis-specification has the greatest impact on power for moderate
$D(a,\omega)$ while mis-specification has scarcely any impact for large or
small values of $D(a,w)$ where all powers converge to 1 or $\alpha$,
respectively. It seems prudent, therefore, to choose a middling value such as
$w=1/2$ if $\omega$ is not known but $H_{4}$ is suspected to avoid the worst
impacts of mis-specification when using $HR\left(\mathbf{p};w\right)$

###  4.3 Case 3: incorrect hypothesis, correctly specified $w$

Assuming all $p_{i}$ are non-null is not always appropriate, however. The
investigations of the previous sections are a useful benchmark and exploration
of the impact of mi-specification, but do not address the natural case of
$H_{3}$ when a handful of significant variables are assumed to exist in a host
of insignificant ones. We therefore repeat the benchmark experiment of Section
4.1 under $H_{3}$ by generating 10,000 independent samples $\mathbf{p}_{i}$ of
size $M=10$ with the first $M\eta$ distributed according to
$f=Beta(a,1/w+a(1-1/w))$ and the latter $M(1-\eta)$ distributed according to
$U$ for $\eta\in\\{0,0.1,\dots,1\\}$ under each of the settings explored
previously. As $HR\left(\mathbf{p};w\right)$ is symmetric in all of its
arguments, this gives no loss of generality. Figure 5 displays contours of the
power surface, the proportion of correct rejections of $H_{0}$, as a function
of $\eta$ and $D(a,w)$ facetted by $\ln(w)$.

Figure 5: The power of $HR\left(\mathbf{p};w\right)$ under $H_{3}$ by $D(a,w)$
and $\eta$ split by $\ln(w)$. Note how the contour for a power of one extends
nearly to the bottom of the plot when $w=1$, but stops near 0.75 when
$w=e^{-6}$.

Figure 5 places the measure of strength of evidence horizontally and the
measure of prevalence vertically. Including $\eta=0$ captures the behaviour of
$HR\left(\mathbf{p};w\right)$ under $H_{0}$ along the bottom edge of the power
surface and including $\eta=1$ captures its behaviour under $H_{4}$ along the
top edge for a range of KL divergences. In particular, this means the top left
part of each subplot corresponds to a generative process for $\mathbf{p}$ with
relatively weak evidence in all $p$-values of $\mathbf{p}$ and the bottom
right part corresponds to a generative process with strong evidence
concentrated in a small number of $p$-values. Figures 3 and 4 show that
rejection occurs at a rate of $\alpha$ once the evidence is weak enough, so
the top left corner is less interesting than the top centre, which gives the
power for tests of moderate strength spread throughout $\mathbf{p}$.

When $w$ is small and $l_{HR}(\mathbf{p};w)\approx l_{Pea}(\mathbf{p})/2$,
$HR\left(\mathbf{p};w\right)$ is relatively weak when strong evidence is
concentrated in a few tests. The contours for $w=e^{-6}$ are nearly horizontal
for log KL divergences between 2.5 and 5 and the power is nearly identical in
the bottom right and the top left corners. On the other hand, when $w\approx
1$ and $l_{w}(\mathbf{p})\approx-l_{Fis}(\mathbf{p})/2$,
$HR\left(\mathbf{p};w\right)$ is relatively powerful when evidence is strong
and concentrated in a few tests. This is starkly visible when $w=1$, the power
contours are nearly vertical, and the bottom right corner has a power of
almost 1. Between these extremes, the power contours display a mix of these
seemingly oppositional sensitivities to strength and prevalence.

###  4.4 Case 4: when both the hypothesis and $w$ are incorrect

Finally, consider the most pessimistic case. In the preceding section, $w$ at
least matched the non-null distribution $f$ under $H_{3}$, but this may not
always be so. Suppose, instead, everything is mis-specified. That is, generate
$M\eta$ $p$-values according to $f=Beta(a,1/\omega+a(1-1/\omega))$ and
$M(1-\eta)$ from the uniform distribution and compute
$HR\left(\mathbf{p};w\right)$ for each of $w=e^{-6},e^{-3},1/2,$ and $1$ for
every setting from Section 4.3 and repeat this generation 10,000 times. The
power, given by proportion of rejections, follows an interesting pattern in
the strength and prevalence of evidence, which is illustrated by the power
contours of $HR\left(\mathbf{p};1\right)$ in Figure 6 and the difference in
power contours in Figure 7.

Figure 6: Power contours of $HR\left(\mathbf{p};1\right)$ under $H_{3}$ by
$D(a,\omega)$ and $\eta$ displayed using the saturation palette of Figure 5.
Figure 7: Contours of the difference in power of $HR\left(\mathbf{p};1\right)$
and $HR\left(\mathbf{p};\omega\right)$ under $H_{3}$ by $D(a,w)$ and $\eta$,
displayed using a divergent palette. $HR\left(\mathbf{p};1\right)$ is more
powerful for strong evidence concentrated in a few tests and less powerful
when weak evidence is spread among most tests.

Compared to the power contours of $HR\left(\mathbf{p};\omega\right)$ in Figure
5, the contours of $HR\left(\mathbf{p};1\right)$ in Figure 6 show greater red
saturation – and thus greater power – in the lower right corner of every
panel. Thus, $HR\left(\mathbf{p};1\right)$ has power greater than or equal to
that of $HR\left(\mathbf{p};\omega\right)$ in this region for every $\omega$.
The magnitude of this improvement is unclear, however, due to the difficulty
comparing the contours between plots. Figure 7 facilitates the comparison by
plotting the difference between the power contours of
$HR\left(\mathbf{p};1\right)$ and $HR\left(\mathbf{p};\omega\right)$ directly
for all settings. This more precise plot demonstrates that the most powerful
$HR\left(\mathbf{p};\omega\right)$ to test $H_{3}$ against $H_{0}$ depends on
the the strength and prevalence of evidence alone, _not_ $\omega$.

Specifically, Figure 7 shows that $HR\left(\mathbf{p};1\right)$ is more
powerful than the correctly-specified $HR\left(\mathbf{p};\omega\right)$ in
the lower right corner of the $D(a,\omega)$, $\eta$ space and does worse left
of centre at the top for both $\omega=e^{-6}$ and $e^{-3}$. In the final panel
where $\omega=1$,
$HR\left(\mathbf{p};1\right)=HR\left(\mathbf{p};\omega\right)$ and so the
difference in their powers is zero everywhere. Recalling the interpretation of
these regions for these facets, this indicates $HR\left(\mathbf{p};1\right)$
is more powerful than $HR\left(\mathbf{p};\omega\right)$ at testing $H_{3}$
against $H_{0}$ when strong evidence is concentrated in a few tests, but is
less powerful when evidence is weaker and spread widely. This pattern holds
for every $\omega$, albeit with differences in magnitude. Additionally, it is
not symmetric: when $HR\left(\mathbf{p};1\right)$ is more powerful, the
magnitude of the difference tends to be greater than in regions where it is
less powerful.

This parallels Loughin (2004), who found $Fis(\mathbf{p})$ more powerful than
other alternatives against $H_{3}$ when strong evidence was concentrated in a
few tests. As $l_{Fis}(\mathbf{p})\propto l_{HR}(\mathbf{p};1)\implies
Fis(\mathbf{p})=HR\left(\mathbf{p};1\right)$, this means that Fisher’s method
is once again relatively powerful for the same setting among the UMP family of
tests $HR\left(\mathbf{p};\omega\right)$. The consistency of this result in
both simulation studies warrants further investigation. To aid in this,
marginal and central rejection levels are introduced to capture the tendency
of a test to reject weak evidence spread among all tests and strong evidence
concentrated in a few, along with a meaningful quotient that combines the
them.

##  5 Central and marginal rejection levels in pooled $p$-values

Recall that any pooled $p$-value $g(\mathbf{p})$ behaves like a univariate
$p$-value, that is $g(\mathbf{p})\in[0,1]$ and under $H_{0}$,
$g(\mathbf{p})\sim U$. It may be based on order statistics and use Equation
(3),

$ord\left(\mathbf{p};k\right)=\sum_{l=k}^{M}{M\choose
l}p_{(k)}^{l}(1-p_{(k)})^{M-l},$

or it may be based on the quantile transformations of Equation (4),

$g(\mathbf{p})=1-F_{M}\left(\sum_{i=1}^{M}F^{-1}(1-p_{i})\right).$

In either case, $g(\mathbf{p})$ must be non-decreasing in every argument if it
is to create convex acceptance regions and therefore be admissible777See
Birnbaum (1954) and Owen (2009) for a detailed discussion of admissibility and
convexity., continuous in every $p_{i}$ if it is to have well-defined
rejection boundaries, and symmetric in $p_{i}$ if no margin is to be favoured.
Given these common properties, concepts of marginal and central rejection can
be defined in order to describe rejection in the cases of evidence against
$H_{0}$ concentrated in one test and evidence against $H_{0}$ spread among all
tests, respectively.

These correspond to the largest $p$-values for a given FWER $\alpha$ which
still result in rejection in two separate cases. The first of these, the
marginal level of the pooled $p$-value, is the largest value of the minimum
$p$-value which still leads to rejection at level $\alpha$. The second, the
central level of the pooled $p$-value, is the largest value which all
$p$-values can take simultaneously while still resulting in rejection.

###  5.1 Characterizing central behaviour

The simulations of Section 4 suggest a pooled $p$-value $g(\mathbf{p})$ which
is powerful at rejecting weak evidence spread among all $p$-values will reject
$H_{0}$ when all tests give relatively large $p$-values compared to $\alpha$.
Under the rejection rule $g(\mathbf{p})\leq\alpha$, this is captured by the
largest $p_{c}$ such that $g(\mathbf{p}_{c})=\alpha$ for
$\mathbf{p}_{c}={(p_{c},\dots,p_{c})}^{\mkern-1.5mu\mathsf{T}}$. Explicitly:

######  Definition 1 (The central rejection level).

For a pooled $p$-value $g(\mathbf{p})$, the central rejection level, $p_{c}$,
is the largest $p$-value for which $g(\mathbf{p})\leq\alpha$ when
$\mathbf{p}={(p_{c},\dots,p_{c})}^{\mkern-1.5mu\mathsf{T}}$. That is

$p_{c}(g)=\sup\big{\\{}p\in[0,1]:g(p,p,\dots,p)\leq\alpha\big{\\}}$ (8)

$p_{c}$ quantifies the maximum $p$-value shared by all tests which still leads
to rejection and therefore is directly related to the power of $g(\mathbf{p})$
along $\mathbf{p}_{c}$. As $g(\mathbf{p})$ is non-decreasing, rejection occurs
for any $\mathbf{p}$ in the hypercube $[0,p_{c}]^{M}$ and so a larger $p_{c}$
implies rejection for a larger volume of $[0,1]^{M}$. It also suggests pooled
$p$-values smaller than $p_{c}$ within this hypercube if $g(\mathbf{p})$ is
continuous and monotonic.

This general definition can be applied to the pooled $p$-values of Equations
(3) and (4) to obtain simple expressions for $p_{c}$.

######  Proposition 1 (The central rejection level of order statistics).

$p_{c}(ord\left(\mathbf{p};k\right))$ is given by the largest $p\in[0,1]$
which satisfies

$\sum_{l=k}^{M}{M\choose l}p^{l}(1-p)^{M-l}\leq\alpha.$ (9)

###### Proof.

The definition of $p_{c}$ forces $p_{(1)}=p_{(2)}=\dots=p_{(M)}=p_{c}$.
Therefore $p_{(k)}=p_{c}$ and so $p_{c}$ is the largest $p\in[0,1]$ which
satisfies $ord\left((p,\dots,p);k\right)\leq\alpha$. Expanding
$ord\left((p,\dots,p);k\right)$ gives Equation (9). ∎

While this cannot generally be solved for a closed-form $p_{c}$, the
particular cases of $ord\left(\mathbf{p};1\right)=Tip(\mathbf{p})$ and
$ord\left(\mathbf{p};M\right)$ admit

$p_{c}(Tip)=1-(1-\alpha)^{\frac{1}{M}}$ (10)

and

$p_{c}(ord\left(;M\right))=\alpha^{\frac{1}{M}}$ (11)

respectively. Also note that $p_{c}(ord\left(;k\right))$ defines a constant
rejection boundary around the regions of $[0,1]^{M}$ where $k-1$ $p$-values
are less than or equal to $p_{c}$, $M-k$ elements are greater, and exactly one
is equal to $p_{c}$. In particular, this means that
$p_{c}(Tip)=p_{c}(ord\left(;1\right))$ is constant along each margin, as $M-1$
points are greater than $p_{c}(Tip)$ along each margin. Next, consider $p_{c}$
for the general quantile transformation of Equation 4.

######  Proposition 2 (The central rejection level of quantile pooled
$p$-values).

Given a pooled $p$-value based on quantile transformations as in Equation (4),

$g(\mathbf{p})=1-F_{M}\left(\sum_{i=1}^{M}c_{i}F^{-1}(1-p_{i})\right),$

the central rejection level is given by

$p_{c}(g(\mathbf{p}))=1-F\left(\frac{1}{\sum_{i=1}^{M}c_{i}}F_{M}^{-1}(1-\alpha)\right)$
(12)

if $F$ and $F_{M}$ are continuous CDFs.

###### Proof.

As $F$ and $F_{M}$ are CDFs , they are monotonically non-decreasing real-
valued functions over their ranges. If $F$ and $F_{M}$ are also continuous,
then $F^{-1}$ and $F_{M}^{-1}$ are continuous. Therefore, we can drop the
supremum from Equation 8 and consider the equality

$\alpha=g(p_{c},p_{c},\dots,p_{c}).$

Expanding $g$ and solving for $p_{c}$ implies

$p_{c}=1-F\left(\frac{1}{\sum_{i=1}^{M}c_{i}}F_{M}^{-1}(1-\alpha)\right).$

∎

If the $p$-values are unweighted, then $c_{1}=c_{2}=\dots=c_{M}=1$,
$\sum_{i=1}^{M}c_{i}=M$ and the behaviour of $p_{c}$ depends on the relative
growth of $F_{M}^{-1}$ in $M$. If $F_{M}^{-1}(1-\alpha)$ grows in $M$ such
that $\frac{1}{M}F^{-1}_{M}(1-\alpha)$ is unbounded, $p_{c}$ will go to zero.
If, on the other hand,
$\lim_{M\rightarrow\infty}\frac{1}{M}F^{-1}_{M}(1-\alpha)=c<\infty$, $p_{c}$
will go to $1-F(c)$. This provides a general expression for the soft
truncation threshold of Zaykin et al. (2007) and suggests interesting
asymptotic behaviour for pooled $p$-values based on quantile functions along
the line $p_{1}=p_{2}=\dots=p_{M}$.

This behaviour can be demonstrated concretely for several quantile
transformations. Stouffer et al. (1949) takes $F=\Phi$ and
$F_{M}=\sqrt{M}\Phi$888Though it uses scaling in computation to avoid this,
the distribution is the same. in Equation 4 to give $Sto(\mathbf{p})$ and so

$\lim_{M\rightarrow\infty}\frac{1}{M}F^{-1}_{M}(1-\alpha)=\lim_{M\rightarrow\infty}\frac{1}{\sqrt{M}}\Phi^{-1}(1-\alpha)=0$

for all $\alpha>0$. This implies

$\lim_{M\rightarrow\infty}p_{c}(Sto)=1-\Phi(0)=\frac{1}{2}.$ (13)

Indeed, taking $Sto(\mathbf{p})$, substituting $p_{1}=\dots=p_{M}=p_{c}(Sto)$,
and taking the limit gives

$\lim_{M\rightarrow\infty}1-\Phi\left(\frac{1}{\sqrt{M}}\sum_{i=1}^{M}\Phi^{-1}(1-p_{c}(Sto))\right)=1-\Phi\left(\lim_{M\rightarrow\infty}\sqrt{M}\Phi^{-1}(1-p_{c}(Sto))\right).$

Evaluating further gives

$\lim_{M\rightarrow\infty}\sqrt{M}\Phi^{-1}(1-p_{c})=\begin{cases}-\infty&\text{
when }p_{c}>\frac{1}{2}\\\ 0&\text{ when }p_{c}=\frac{1}{2}\\\ \infty&\text{
when }p_{c}<\frac{1}{2},\end{cases}$

and so $Sto(\mathbf{p})$ is either $0$, $\frac{1}{2}$, or $1$ for large $M$
when $p_{1}\approx p_{2}\approx\dots\approx p_{M}$. Furthermore, it will
reject $H_{0}$ for _any_ FWER level $\alpha$ if $p_{1},p_{2},\dots,p_{M}$ are
all less than $\frac{1}{2}$ when $M$ is large enough.

$F_{\chi}(x;\kappa)$ admits similar analysis. By the central limit theorem

$\lim_{\kappa\rightarrow\infty}\chi^{2}_{\kappa}\rightarrow Z\sim
N(\kappa,2\kappa)$

where $N(\mu,\sigma^{2})$ is a normal distribution with mean $\mu$ and
variance $\sigma^{2}$. Therefore, as $M\rightarrow\infty$

$F_{\chi}(x;M\kappa)\rightarrow\Phi\left(\frac{x-M\kappa}{\sqrt{2M\kappa}}\right).$

This implies that the pooled $p$-value based on the $\chi^{2}$ quantile has
the limiting value

$\lim_{M\rightarrow\infty}1-F_{\chi}\Big{(}MF_{\chi}^{-1}\big{(}1-p_{c};\kappa\big{)};M\kappa\Big{)}=1-\lim_{M\rightarrow\infty}\Phi\left(\frac{MF_{\chi}^{-1}\big{(}1-p_{c};\kappa\big{)}-M\kappa}{\sqrt{2M\kappa}}\right)$

when $p_{1}=\dots=p_{M}=p_{c}$. As $\Phi$ is absolutely continuous the limit
can be taken inside the argument to give

$1-\Phi\left(\lim_{M\rightarrow\infty}\sqrt{\frac{M}{2\kappa}}\left[F_{\chi}^{-1}(1-p_{c};\kappa)-\kappa\right]\right).$

Now,

$\lim_{M\rightarrow\infty}\sqrt{\frac{M}{2\kappa}}\left[F_{\chi}^{-1}(1-p_{c};\kappa)-\kappa\right]=\begin{cases}-\infty&\text{
when }p_{c}>1-F_{\chi}(\kappa;\kappa)\\\ 0&\text{ when
}p_{c}=1-F_{\chi}(\kappa;\kappa)\\\ \infty&\text{ when
}p_{c}<1-F_{\chi}(\kappa;\kappa),\end{cases}$

and so the pooled $p$-value based on the $\chi^{2}$ quantile transformation
behaves similarly to $Sto(\mathbf{p})$. It is asymptotically either 1,
$\frac{1}{2}$, or 0 when $p_{1}\approx p_{2}\approx\dots\approx p_{M}$
depending on whether all are greater than, equal to, or less than
$1-F_{\chi}(\kappa;\kappa)$. Additionally, this implies that

$p_{c}=1-F_{\chi}(\kappa;\kappa)$ (14)

for the $\chi^{2}$ quantile case. So, although $p_{c}$ depends on $\kappa$,
all $\chi^{2}$ quantile pooled $p$-values have identical behaviour about their
respective $p_{c}$.

The form of Equation (4) suggests that this result applies generally. Suppose
the random variable with CDF $F$ has a mean $\mu$ and variance $\sigma^{2}$.
Under $H_{0}$, $F^{-1}(1-p_{i})$ for $i=1,\dots,M$ are independent and
identically-distributed realizations of this random variable and so
$\sum_{i=1}^{M}F^{-1}(1-p_{i})$ is asymptotically normally distributed with
mean $M\mu$ and variance $M\sigma^{2}$ by the central limit theorem. Therefore
$F_{M}\xrightarrow[M\rightarrow\infty]{}\sqrt{M}\sigma\Phi+M\mu$ for the
pooled $p$-value based on $F$ and the harsh asymptotic boundary at $p_{c}$
derived for $Sto$ will occur for _any_ evidential statistic that uses quantile
functions.

###  5.2 Characterizing marginal behaviour

Besides the central behaviour of a pooled $p$-value $g(\mathbf{p})$, the
simulations of Section 4 indicate large differences in power occur for the
rejection rule $g(\mathbf{p})\leq\alpha$ when strong evidence exists in a
single test. This is captured by the _marginal rejection level at b_.

######  Definition 2 (The marginal rejection level at $b$).

For a symmetric pooled $p$-value $g(\mathbf{p})$, the marginal rejection level
at $b$, $p_{r}(g;b)$, is the largest individual $p$-value in $[0,b]$ for which
$g(\mathbf{p})\leq\alpha$ when all other $p$-values are $b\in[0,1]$. Without
loss of generality, define

$p_{r}(g;b)=\sup\big{\\{}p_{1}\in[0,b]:g(p_{1},b,\dots,b)\leq\alpha\big{\\}}.$
(15)

In particular, the marginal value when $b=1$ is of interest, that is when
there is minimal evidence against all hypotheses other than $H_{01}$.
Therefore, also define

$p_{r}(g)=\lim_{b\rightarrow
1}\sup\big{\\{}p_{1}\in[0,b]:g(p_{1},b,\dots,b)\leq\alpha\big{\\}}.$ (16)

Note that symmetry is only necessary to avoid defining marginal rejection
levels for each index $i\in\\{1,\dots,M\\}$ separately and that the term
_marginal rejection level_ refers to Equation (16). If $g$ is non-decreasing
in all of its arguments, $p_{r}(g;b)$ gives the largest value of $p_{(1)}$
that still leads to rejection at $\alpha$ when the evidence in all other
$p$-values is bounded at $b$. The most extreme version of this measure is
given by $p_{r}(g)$. By taking $b=1$, it measures the power of $g$ for
evidence in a single test when all other tests provide no evidence against
$H_{0}$, and so the sensitivity of $g$ to evidence in a single test. This
leads to a key lemma for $ord\left(\mathbf{p};1\right)=Tip(\mathbf{p})$.

######  Lemma 1 (The marginal rejection level for the minimum statistic).

The marginal rejection level for $g_{Tip}$ has two cases:

$p_{r}(Tip;b)=\begin{cases}b&\text{ for }b<1-(1-\alpha)^{\frac{1}{M}}\\\
1-(1-\alpha)^{\frac{1}{M}}&\text{ for }b\geq
1-(1-\alpha)^{\frac{1}{M}}.\end{cases}$

###### Proof.

Recall that

$Tip(\mathbf{p})=1-(1-p_{(1)})^{M}$

is a function of the minimum alone. Rejection occurs when
$Tip(\mathbf{p})\leq\alpha$, or rather when $p_{(1)}\leq
1-(1-\alpha)^{\frac{1}{M}}$. When $b<1-(1-\alpha)^{\frac{1}{M}}$ and
$p_{1}\leq b$, all values are below the rejection threshold and so $p_{(1)}$
attains its upper bound. Therefore

$p_{r}(Tip;b)=\sup\big{\\{}p_{1}\in[0,b]:g(p_{1},b,\dots,b)\leq\alpha\big{\\}}=b.$

When $b\geq 1-(1-\alpha)^{\frac{1}{M}}$, rejection will only occur if
$p_{(1)}$ is below the rejection threshold at $\alpha$, and so

$p_{r}(Tip;b)=\sup\big{\\{}p_{1}\in[0,b]:g(p_{1},b,\dots,b)\leq\alpha\big{\\}}=1-(1-\alpha)^{\frac{1}{M}}.$

∎

A direct consequence of Lemma 1 is that $p_{r}(g_{Tip})=p_{c}(g_{Tip})$, which
Theorem 1 proves is uniquely true for $Tip(\mathbf{p})$.

As with $p_{c}$, a larger $p_{r}$ indicates greater power in a particular
region of the unit hypercube. While $p_{c}$ defines the rejection cube
$[0,p_{c}]^{M}$, $p_{r}$ defines the rejection shell
$\\{\mathbf{p}\in[0,1]^{M}:p_{(1)}\leq p_{r}\\}$ with a flat boundary at
$p_{r}$ along each margin. A larger $p_{r}$ implies a larger shell and
therefore a greater volume of $[0,1]^{M}$ where $H_{0}$ is rejected and
smaller pooled $p$-values within this volume if $g(\mathbf{p})$ is monotonic.
Again, general expressions are provided for $p_{r}$ for the order statistic
and quantile transformation pooled $p$-values.

######  Proposition 3 (The marginal rejection level for order statistics).

For $k\geq 2$, $p_{r}(ord\left(;k\right),b)=b$ when $\sum_{l=k}^{M}{M\choose
l}b^{l}(1-b)^{M-1}\leq\alpha$ and does not exist otherwise.

###### Proof.

Recall that

$g_{Ord}(\mathbf{p};k)=\sum_{l=k}^{M}{M\choose l}p_{(k)}^{l}(1-p_{(k)})^{M-l}$

and note Equation 15 forces $p_{(k)}=b$ for all $k>1$. If
$\sum_{l=k}^{M}{M\choose l}b^{l}(1-b)^{M-1}\leq\alpha$, then the supremum of
$p_{1}$ is $b$. On the other hand, if $\sum_{l=k}^{M}{M\choose
l}b^{l}(1-b)^{M-1}\geq\alpha$ there is no value of $p_{1}$ which leads to
rejection and so $p_{r}(ord\left(;k\right),b)$ does not exist. ∎

In particular, this implies that $p_{r}(ord\left(;k\right))$ does not exist
for $k\geq 2$, in other words the pooled $p$-value based on $p_{(k)}$ has a
value independent of $p_{(1)}$ for $k\geq 2$. So long as $k$ tests are less
than a particular bound, $ord\left(\mathbf{p};k\right)$ will reject. If fewer
than $k$ are below that bound, the values of these small $p$-values are
irrelevant.

######  Proposition 4 (The marginal rejection level of quantile transformation
statistics).

Given an unweighted evidential statistic based on quantile transformations as
in Equation 4,

$g(\mathbf{p})=1-F_{M}\left(\sum_{i=1}^{M}F^{-1}(1-p_{i})\right),$

if $F_{M}$ and $F$ are both continuous then

$p_{r}(g;b)=1-F\Big{(}F_{M}^{-1}(1-\alpha)-[M-1]F^{-1}(1-b)\Big{)}.$

Further, if both are absolutely continuous

$p_{r}(g)=1-F\left(F_{M}^{-1}(1-\alpha)-[M-1]\lim_{x\rightarrow
0+}F^{-1}(x)\right)$ (17)

###### Proof.

Substituting Equation 4 into Equation 15 gives

$p_{r}(g;b)=\sup\Bigg{\\{}p:1-F_{M}\bigg{(}F^{-1}(1-p)+[M-1]F^{-1}(1-b)\bigg{)}\leq\alpha\Bigg{\\}}.$

As both $F_{M}$ and $F$ are CDFs, they are non-decreasing, if they are also
continuous their inverses exist and the supremum can be dropped to give

$p_{r}(g;b)=1-F\Big{(}F_{M}^{-1}(1-\alpha)-[M-1]F^{-1}(1-b)\Big{)}.$

If they are both absolutely continuous, then the limit

$1-\lim_{b\rightarrow 1}F\Big{(}F_{M}^{-1}(1-\alpha)-(M-1)F^{-1}(1-b)\Big{)}$

can be taken into the argument of $F$ to give Equation 17. ∎

Many proposals use absolutely continuous CDFs, so this can be readily applied.
$Sto(\mathbf{p})$, for example, has

$p_{r}(Sto)=1-\Phi\left(\sqrt{M}\Phi^{-1}(1-\alpha)-[M-1]\lim_{x\rightarrow
0+}\Phi^{-1}(x)\right)=0$

for any $\alpha$ and $M$ as $\lim_{x\rightarrow 0}\Phi(x)=-\infty$. Similarly,
the proposal by Mudholkar and George (1977) has $p_{r}=0$, as it uses the
logistic distribution which also has $\lim_{x\rightarrow 0}F(x)=-\infty$. This
suggests that, for a large enough $p$-value on all remaining tests, no level
of evidence in a single test will cause the rejection of $H_{0}$ for either of
these pooled $p$-values; their marginal rejection levels are always 0 for
large enough $b$.

###  5.3 The centrality quotient

Beyond providing definitions that clarify the power of a pooled $p$-value to
detect evidence spread among all tests and evidence in a single test, $p_{c}$
and $p_{r}$ as defined in Equations (16) and (8) can be combined into a single
value summarizing the relative preference for diffuse or concentrated
evidence. First, a key relationship between $p_{c}$ and $p_{r}$ is proven.

######  Theorem 1 (Order of $p_{c}$ and $p_{r}$).

For a pooled $p$-value $g(\mathbf{p})$ that is continuous, symmetric, and
monotonically non-decreasing in all arguments, $p_{c}\geq p_{r}$ if both
exist. Furthermore, equality occurs iff $g(\mathbf{p})$ is constant in $p_{k}$
for $p_{k}\neq p_{(1)}$, that is if $g(\mathbf{p})=f(p_{(1)})$ is a function
of the minimum $p$-value alone.

###### Proof.

Consider $p_{c}(g)$ as in Definition 8. Then

$p_{c}(g)=\sup\big{\\{}p\in[0,1]:g(p,\dots,p)\leq\alpha\big{\\}}.$

Suppose

$p_{r}(g)=\lim_{b\rightarrow
1}\sup\big{\\{}p_{1}\in[0,b]:g(p_{1},b,\dots,b)\leq\alpha\big{\\}}$

exists. If $g$ is symmetric, $p_{r}(g)$ captures the marginal rejection level
of $g$ in all margins. If $g$ is continuous, then both $p_{c}(g)$ and
$p_{r}(g)$ lie on the $\alpha$ level surface of $g$. Therefore

$g(p_{c},\dots,p_{c})=\alpha=g(p_{r},1,\dots,1).$

But $g$ is non-decreasing in all of its arguments, so

$g(p_{c},1,\dots,1)\geq g(p_{c},p_{c},\dots,p_{c})=g(p_{r},1,\dots,1)$

and therefore

$p_{c}\geq p_{r}.$

If $p_{c}=p_{r}$, then substitute

$\alpha=g(p_{c},\dots,p_{c})=g(p_{r},\dots,p_{r}).$

As $g$ is non-decreasing

$g(p_{r},\dots,p_{r})\leq g(p_{r},1,\dots,1)=\alpha$

and so

$g(p_{r},1,\dots,1)=g(p_{r},p_{r},\dots,p_{r}).$

This implies that the average slope of $g$ over $[p_{r},1]$, equivalently
$[p_{c},1]$, is zero for all $p_{k}\neq p_{1}$. As $g$ is continuous and non-
decreasing, this implies that the slope must be zero for every point in this
interval for all $p_{k}\neq p_{1}$. As $p_{k}\geq p_{1}$ for all
$p_{k}\in\mathbf{p}$ over this region, $p_{1}=p_{(1)}$ by definition. By the
symmetry of $g$, the same argument holds for every $p_{k}$. Therefore
$g(\mathbf{p})=f(p_{(1)})$ for some non-decreasing function $f$.

To prove the reverse direction note that if $g(\mathbf{p})=f(p_{(1)})$, then

$\alpha=g(p_{c},p_{c},\dots,p_{c})=g(p_{c},1,\dots,1)$

and so $p_{c}=p_{r}$ by the definition of $p_{r}$ and the continuity of $g$.
By symmetry, this same argument holds for any margin. ∎

Two facts follow directly from this proof. First, Theorem 1 implies that
$p_{c}=p_{r}$ only for $Tip(\mathbf{p})=ord\left(\mathbf{p};1\right)$ among
symmetric, continuous, monotonically non-decreasing $p$-values as
$Tip(\mathbf{p})$ is the unique pooled $p$-value defined by $p_{(1)}$. A
second corollary is the existence of a sensible _centrality quotient_ to
quantify the balance between central and marginal rejection levels in pooled
$p$-values.

######  Definition 3 (The centrality quotient).

Suppose $g$ is a continuous, symmetric, and monotonically non-decreasing
pooled $p$-value for which $p_{r}(g)$ and $p_{c}(g)$ defined as in Equations
(16) and (8) exist, define the centrality quotient

$q(g)=\frac{p_{c}(g)-p_{r}(g)}{p_{c}(g)}.$ (18)

Theorem 1 implies that $q(g)\in[0,1]$ with meaningful bounds. If $q(g)=0$,
$g(\mathbf{p})$ will reject based on the smallest $p$-values alone, increasing
the marginal rejection level as large as possible while staying non-
decreasing. Moreover, $q(g)=0$ implies $g(\mathbf{p})$ is the pooled $p$-value
based on $p_{(1)}$ alone, $Tip(\mathbf{p})$. In contrast, when $q(g)=1$, $g$
cannot reject based on the evidence contained in a single test, instead it
requires evidence in many or all tests, for example $Sto(\mathbf{p})$. Between
these extremes, pooled $p$-values with larger centrality quotients will reject
$H_{0}$ for a larger range of $p_{c}$ values and a smaller range of $p_{r}$
values, and so will be more powerful at detecting evidence spread broadly at
the cost of power when evidence is concentrated in a small number of
$p$-values.

Indeed, increasing $w$ decreases the centrality quotient of
$HR\left(\mathbf{p};w\right)$. This matches the empirical results obtained in
Section 4.4 and in particular Figure 4.4, where larger $w$ values provided
greater power when the prevalence of evidence was small but the strength of
evidence was large and smaller $w$ values gave greater power in the case of
weak evidence with high prevalence. For $\alpha=0.05$, the centrality
quotients of a range of $w$ values in $HR\left(\mathbf{p};w\right)$ are
compared to those of several quantile transformation proposals in Table 1 over
a range of $M$ values. As predicted by the asymptotic argument at the end of
Section 5.1, every method tends towards a centrality of 1 as $M$ increases and
each $F_{M}$ converges to a corresponding normal CDF.

| $M$
---|---
Pooled $p$-value | 2 | 5 | 10 | 20
Tippett (1931) | 0 | 0 | 0 | 0
Cinar and Viechtbauer (2022) | 0.83 | 0.99 | 1.00 | 1.00
Stouffer et al. (1949) | 1 | 1 | 1 | 1
Fisher (1932) | 0.91 | 1.00 | 1.00 | 1.00
Mudholkar and George (1977) | 1 | 1 | 1 | 1
Wilson (2019) | 0.49 | 0.79 | 0.90 | 0.95
$HR\left(\mathbf{p};e^{-6}\right)$ | 1.00 | 1.00 | 1.00 | 1.00
$HR\left(\mathbf{p};e^{-3}\right)$ | 1.00 | 1.00 | 1.00 | 1.00
$HR\left(\mathbf{p};1\right)$ | 0.91 | 1.00 | 1.00 | 1.00
Table 1: Centrality quotients for certain pooled $p$-values.

Beyond $HR\left(\mathbf{p};w\right)$, other methods show the same relationship
between the centrality quotient and regions of relative power in the empirical
explorations in Westberg (1985), Loughin (2004), and Kocak (2017). Pooled
$p$-values with larger centrality quotients are more powerful for weak
evidence spread among all tests than those with smaller centrality quotients,
but are relatively weak against strong evidence concentrated in a few tests.
This is suggestive of an inverse relationship between $p_{r}$ and $p_{c}$ over
different pooled $p$-values, but this is not the case generally. Consider, as
a counter-example, $Sto(\mathbf{p})$ and the proposal of Mudholkar and George
(1977): both have $p_{r}=0$ but different values of $p_{c}$.

##  6 Controlling the centrality quotient

Table 1, and others which could be constructed like it, provide only a limited
ability to select a centrality quotient. Most of the proposals have centrality
near 1, and all proposals approach 1 as $M$ increases. Rather than choose
among these other limited proposals when power is desired in a particular
region, this work proposes a family of quantile pooled $p$-values based on
$\chi^{2}_{\kappa}$ which precisely controls the centrality for any $M$.
Following Equation (4), define the $\chi^{2}_{\kappa}$ quantile pooled
$p$-value

$chi\left(\mathbf{p};\kappa\right)=1-F_{\chi}\left(\sum_{i=1}^{M}F_{\chi}^{-1}(1-p_{i};\kappa);M\kappa\right)$
(19)

where $\kappa\in[0,\infty)$ is the the degrees of freedom of the quantile
transformation applied to the $p_{i}$ and doubles as a centrality parameter
that sets $q(chi\left(;\kappa\right))$ arbitrarily.999This is similar to the
gamma method of Zaykin et al. (2007), but with a different parameter choice.
It is possible the same control of $c$ may be obtained with a general gamma
CDF, but sticking to the $\chi^{2}_{\kappa}$ simplifies the number of
parameters from two to one. This family of pooled $p$-values includes several
widely-used previous proposals. Setting $\kappa=2$ gives $Fis(\mathbf{p})$,
$\kappa=1$ gives the proposal from Cinar and Viechtbauer (2022), taking
$\lim_{\kappa\rightarrow\infty}chi\left(\mathbf{p};\kappa\right)$ gives
$Sto(\mathbf{p})$, and taking $\lim_{\kappa\rightarrow
0}chi\left(\mathbf{p};\kappa\right)$ gives $Tip(\mathbf{p})$. While the former
two are by definition, the latter must be proven. First, prove
$\lim_{\kappa\rightarrow\infty}chi\left(\mathbf{p};\kappa\right)=Sto(\mathbf{p})$
by applying the central limit theorem.

######  Theorem 2 (Limiting value of $chi\left(\mathbf{p};\kappa\right))$ as
$\kappa\rightarrow\infty$).

$\lim_{\kappa\rightarrow\infty}chi\left(\mathbf{p};\kappa\right)=Sto(\mathbf{p})$

###### Proof.

Note that Equation (19) is always a pooled $p$-value, i.e. has a uniform
distribution for any choice of $\kappa$. By the CLT,
$\lim_{\kappa\rightarrow\infty}F_{\chi}(x;\kappa)=\Phi(x)$, and so in the
limit $chi\left(\mathbf{p};\kappa\right)$ becomes the pooled $p$-value derived
from the sum of standard normal quantile transformations, $Sto(\mathbf{p})$. ∎

The proof for $chi\left(\mathbf{p};0\right)$ is slightly more involved, and
relies on Theorem 1.

######  Theorem 3 (Limiting value of $chi\left(\mathbf{p};\kappa\right)$ for
$\kappa=0$).

$\lim_{\kappa\rightarrow
0}chi\left(\mathbf{p};\kappa\right)=Tip(\mathbf{p})=ord\left(\mathbf{p};1\right)$

###### Proof.

Theorem 1 proves that $p_{r}=p_{c}$ for a pooled $p$-value if and only if that
pooled $p$-value is $Tip=ord\left(;1\right)$. Therefore, the limit is proven
if

$\lim_{\kappa\rightarrow
0}p_{r}\big{(}chi\left(;\kappa\right)\big{)}=\lim_{\kappa\rightarrow
0}p_{c}\big{(}chi\left(;\kappa\right)\big{)}$

Expressing these quantities as probability statements gives

$p_{c}\big{(}chi\left(;\kappa\right)\big{)}=P\left(\chi^{2}_{\kappa}\geq\frac{1}{M}F_{\chi}^{-1}(1-\alpha;M\kappa)\right),$

and

$p_{r}\big{(}chi\left(;\kappa\right)\big{)}=P\left(\chi^{2}_{\kappa}\geq
F_{\chi}^{-1}(1-\alpha;M\kappa)\right).$

The case of $\chi^{2}_{0}$ is a degenerate distribution at 0. That is

$F_{\chi^{2}}(x;0)=\begin{cases}0&x<0\\\ 1&x\geq 0.\end{cases}$

This can also be seen from the limit of Markov’s inequality for the
$\chi^{2}_{\kappa}$ distribution,

$P(\chi^{2}_{\kappa}\geq a)\leq\frac{\kappa}{a},$

which goes to zero for any $a>0$ as $\kappa\rightarrow 0$. As
$F_{\chi}(x;\kappa)$ is continuous and monotonically increasing for all
$\kappa$, this also implies

$F\left(\frac{\kappa}{\alpha};\kappa\right)\geq 1-\alpha$ $\implies
1-F\left(\frac{\kappa}{\alpha};\kappa\right)\leq\alpha$ $\implies
P\left(\chi^{2}_{\kappa}\geq\frac{\kappa}{\alpha}\right)\leq\alpha$ $\implies
F^{-1}_{\chi}(1-\alpha;\kappa)\leq\frac{\kappa}{\alpha}.$

This bound is not particularly tight, for $\alpha=0.05$ it only restricts the
$0.95$ quantile to be less than 20 times the mean. However, it suffices to
evaluate

$\displaystyle\lim_{\kappa\rightarrow
0}\left|\frac{1}{M}F^{-1}_{\chi}(1-\alpha;M\kappa)-F^{-1}_{\chi}(1-\alpha;M\kappa)\right|$
$\displaystyle=$ $\displaystyle\frac{M-1}{M}\lim_{\kappa\rightarrow
0}F^{-1}_{\chi}(1-\alpha;M\kappa)$ $\displaystyle\leq$
$\displaystyle\frac{M-1}{M}\lim_{\kappa\rightarrow 0}\frac{M\kappa}{\alpha}=0$

and therefore

$\lim_{\kappa\rightarrow
0}\frac{1}{M}F^{-1}_{\chi}(1-\alpha;M\kappa)=\lim_{\kappa\rightarrow
0}F^{-1}_{\chi}(1-\alpha;M\kappa)$

for any $\alpha>0$. This implies that
$p_{c}\big{(}chi\left(;\kappa\right)\big{)}=p_{r}\big{(}chi\left(;\kappa\right)\big{)}$
in the limit $\kappa\rightarrow 0$. ∎

The result of Theorem 3 can be understood intuitively using the non-central
$\chi^{2}_{0}$ of Siegel (1979) with a non-centrality parameter $\lambda$,
call it $\chi^{2}_{0}(\lambda)$. When $\lambda\rightarrow 0$,
$\chi^{2}_{0}(\lambda)\rightarrow\chi^{2}_{0}$ in distribution, so taking
$\chi^{2}_{0}(\lambda)$ with small $\lambda$ should provide some sense of how
$\chi^{2}_{0}$ behaves. Unlike $\chi^{2}_{0}$, however,
$\chi^{2}_{0}(\lambda)$ has a discrete probability mass at 0 for all
$\lambda>0$. As a result, the quantile function of $\chi^{2}_{0}(\lambda)$,
$F^{-1}_{\lambda}$, returns zero for any input less than
$e^{-\frac{\lambda}{2}}$ and so the terms in the sum

$\sum_{i=1}^{M}F_{\lambda}^{-1}(1-p_{i})$

are non-zero only for those $i$ where $p_{i}\leq 1-e^{-\frac{\lambda}{2}}$. As
$\lambda\rightarrow 0$, this sum becomes arbitrarily close to
$chi\left(;0\right)$ but only the smallest $p$-values contribute. Eventually,
only the minimum contributes to the sum, and so
$chi\left(;\kappa\right)\approx f(p_{(1)})$ for very small $\kappa$ values.

The limits $\lim_{\kappa\rightarrow
0}chi\left(\mathbf{p};\kappa\right)=Tip(\mathbf{p})$ and
$\lim_{\kappa\rightarrow\infty}chi\left(\mathbf{p};\kappa\right)=Sto(\mathbf{p})$
are also demonstrated empirically by generating $n_{sim}$ independent
realizations of $\mathbf{p}$ assuming $H_{0}$ is true. For each vector
$\mathbf{p}_{i}$, compute $chi\left(\mathbf{p}_{i};\kappa\right)$ for a range
of $\kappa$, $Sto(\mathbf{p}_{i})$, and $Tip(\mathbf{p}_{i})$ and compare
$chi\left(\mathbf{p}_{i};\kappa\right)$ to the other two pooled $p$-values.
Figure 8 shows this pattern for a few $\kappa$ when $M=5$.

---
(a)
(b)
(c)
Figure 8: A comparison of $chi\left(\mathbf{p}_{i};\kappa\right)$,
$Sto(\mathbf{p}_{i})$, and $Tip(\mathbf{p}_{i})$ values for 1000 independently
generated $\mathbf{p}_{i}\sim Unif([0,1]^{5})$ in the case of (a) small
$\kappa$, (b) moderate $\kappa$, and (c) large $\kappa$.

As expected, the agreement between $chi\left(\mathbf{p};\kappa\right)$ and
$Tip(\mathbf{p})$ is perfect for small enough $\kappa$, the two functions have
identical outputs for $\kappa=0.0035$ in Figure 8(a). Similarly,
$chi\left(\mathbf{p};\kappa\right)$ and $Sto(\mathbf{p})$ match for large
$\kappa$, as when $\kappa\approx 3000$ in Figure 8(c). Note that the
particular values of $\kappa$ where this close agreement occurs will depend on
$M$.

Perhaps more interesting is the curved boundary of the points along the top of
the plot of $chi\left(\mathbf{p};0.0035\right)$ against $Sto(\mathbf{p})$ in
Figure 8(a), many points populate the lower right corner of this plot but
there are none in the upper left. This pattern is mirrored in Figure 8(c) for
the plot of $chi\left(\mathbf{p};2981\right)$ against $Tip(\mathbf{p})$. As
$chi\left(\mathbf{p};\kappa\right)$ is essentially identical to one of
$Tip(\mathbf{p})$ or $Sto(\mathbf{p})$ in these cases, this pattern reflects
the relationship between $Tip(\mathbf{p})$ and $Sto(\mathbf{p})$. By
definition, $Tip(\mathbf{p})$ considers only $p_{(1)}$, but any of the $p_{i}$
can impact $Sto(\mathbf{p})$. As a result there will be many cases where a
small $Tip(\mathbf{p})$ occurs despite a large $Sto(\mathbf{p})$ because a
very small $p_{(1)}$ happens by chance. The reverse is impossible, if
$Tip(\mathbf{p})$ is large then $p_{(1)}$ is large and therefore all values in
$\mathbf{p}$ are large, suggesting a large $Sto(\mathbf{p})$.

###  6.1 Choosing a parameter

In addition to these meaningful limits, there seems to be a monotonically
increasing relationship between $\kappa$ and $q(chi\left(;\kappa\right))$. Let
$\chi^{*}_{\kappa}(\alpha)$ be the $1-\alpha$ quantile of the $\chi^{2}$
distribution with $\kappa$ degrees of freedom, then
$chi\left(\mathbf{p};\kappa\right)$ has the central rejection level

$p_{c}\big{(}chi\left(;\kappa\right)\big{)}=1-F_{\chi}\left(\frac{1}{M}F_{\chi}^{-1}(1-\alpha;M\kappa);\kappa\right)=P\left(\chi^{2}_{\kappa}\geq\frac{1}{M}\chi^{*}_{M\kappa}(\alpha)\right)$
(20)

and the marginal rejection level

$p_{r}\big{(}chi\left(;\kappa\right)\big{)}=1-F_{\chi}\bigg{(}F_{\chi}^{-1}(1-\alpha;M\kappa);\kappa\bigg{)}=P\bigg{(}\chi^{2}_{\kappa}\geq\chi^{*}_{M\kappa}(\alpha)\bigg{)},$
(21)

implying

$\displaystyle q(chi\left(;\kappa\right))$ $\displaystyle=$
$\displaystyle\frac{p_{c}\big{(}chi\left(;\kappa\right)\big{)}-p_{r}\big{(}chi\left(;\kappa\right)\big{)}}{p_{c}\big{(}chi\left(;\kappa\right)\big{)}}$
(22) $\displaystyle=$ $\displaystyle
P\left(\chi^{2}_{\kappa}\leq\chi^{*}_{M\kappa}(\alpha)~{}\vline~{}\chi^{2}_{\kappa}\geq\frac{1}{M}\chi^{*}_{M\kappa}(\alpha)\right).$

That is, the centrality quotient of $chi\left(\mathbf{p};\kappa\right)$ is the
conditional probability that a $\chi^{2}_{\kappa}$ random variable is less
than $\chi^{*}_{M\kappa}(\alpha)$ given that it is greater than
$\frac{1}{M}\chi^{*}_{M\kappa}(\alpha)$.

A better sense of the region corresponding to this conditional probability for
$\alpha<0.5$ is garnered by writing $\chi^{*}_{M\kappa}$ in terms of the mean
of the $\chi^{2}_{M\kappa}$ distribution, $M\kappa$. Taking an arbitrary
remainder function $R_{M\kappa}(\alpha)>0$ such that
$\chi^{*}_{M\kappa}(\alpha):=M\kappa+R_{M\kappa}(\alpha)$, subsitution gives

$q(chi\left(;\kappa\right))=P\left(\chi^{2}_{\kappa}\leq
M\kappa+R_{M\kappa}(\alpha)~{}\vline~{}\chi^{2}_{\kappa}\geq\kappa+\frac{1}{M}R_{M\kappa}(\alpha)\right),$

clarifying that $q(chi\left(;\kappa\right))$ is a conditional probability on
the right tail of the $\chi^{2}_{\kappa}$ distribution when $\alpha<0.5$.
Making more precise statements about $R_{M\kappa}(\alpha)$ is challenging due
to the small values of $\kappa$ which may be chosen for
$chi\left(;\kappa\right)$. Most approximations of $\chi^{2}$ tail
probabilities and quantiles either break down when the degrees of freedom is
less than one or explicitly assume more than one degrees of freedom (Hawkins
and Wixley, 1986; Canal, 2005; Inglot, 2010). Nonetheless, the above
probability can be computed numerically, as was done for the curves of
$q(chi\left(;\kappa\right))$ by $\log_{10}(\kappa)$ for $M$ ranging from 2 to
10,000 in Figure 9.

Figure 9: The centrality quotient of $chi\left(;\kappa\right)$ by
$\log_{10}(\kappa)$

The curves of $q(chi\left(;\kappa\right))$ by $\kappa$ have a consistent
sigmoid shape for all $M$. Most of the change in the centrality quotient
occurs for values in a three unit range in $\log_{10}(\kappa)$ for any $M$,
though the centre of this range decreases as $M$ grows. When $\kappa=10^{-3}$,
for example, the centrality quotient when $M=100$ is greater than 0.8 while
the same $\kappa$ value corresponds with a centrality quotient of less than
0.05 when $M=2$. Just as with any other pooled $p$-value, increasing $M$
increases the centrality of $chi\left(;\kappa\right)$ for a given $\kappa$ as
the sum of independent $p$-values becomes more normally distributed by the
central limit theorem.101010This can also be understood geometrically. For a
pooled $p$-value in $M$ dimensions, the volume of the marginal shell of width
$p_{r}$ is $1-(1-p_{r})^{M}$, which approaches 1 for any $p_{r}>0$ as
$M\rightarrow\infty$. As the total volume of the rejection region is $\alpha$
for the rejection rule $g(\mathbf{p})\leq\alpha$, $p_{r}$ must decrease in $M$
to hold the volume constant.

In practice, the inverse of the above curves may be of greater interest to
control the centrality quotient under $chi\left(\mathbf{p};\kappa\right)$
rather than simply report it. Figure 9 does allow the selection of $\kappa$
for a given centrality quotient by estimating the $\kappa$ value where the
intersection between a curve and a vertical line at $\kappa$ is at the desired
quotient, but a table displaying the numerically estimated inverse for evenly-
spaced $\kappa$ as in Table 2 is more precise and straightforward to use.
Determining the desired $\log_{10}(\kappa)$ for a given centrality quotient
and $M$ proceeds as for a table of critical values. The user searches down the
columns for the $M$ most closely corresponding to the setting at hand, and
then searches through that row for the desired column. If
$q(chi\left(;\kappa\right))=1$ or $0$ is desired, the table is unnecessary as
$Sto(\mathbf{p})$ or $Tip(\mathbf{p})$ can be used directly. Unlike with
critical value tables, there is no need to be conservative: linear
interpolation between the provided $\log_{10}(\kappa)$ values is a reasonable
approach to choosing $\kappa$.

| Centrality quotient
---|---
$M$ | 0.1 | 0.2 | 0.3 | 0.4 | 0.5 | 0.6 | 0.7 | 0.8 | 0.9
2 | -2.1 | -1.7 | -1.4 | -1.2 | -0.9 | -0.7 | -0.4 | -0.1 | 0.3
5 | -2.8 | -2.5 | -2.3 | -2.1 | -1.9 | -1.7 | -1.4 | -1.2 | -0.8
20 | -3.7 | -3.4 | -3.1 | -2.9 | -2.8 | -2.6 | -2.4 | -2.1 | -1.8
100 | -4.6 | -4.3 | -4 | -3.8 | -3.7 | -3.5 | -3.3 | -3 | -2.7
500 | -5.4 | -5.1 | -4.8 | -4.7 | -4.5 | -4.3 | -4.1 | -3.9 | -3.5
2000 | -6.1 | -5.8 | -5.5 | -5.3 | -5.2 | -5 | -4.8 | -4.6 | -4.2
10000 | -6.9 | -6.6 | -6.3 | -6.1 | -6 | -5.8 | -5.6 | -5.4 | -5
Table 2: $\log_{10}(\kappa)$ values by $q(chi\left(;\kappa\right))$ and $M$ to
aid in parameter selection for the desired balance of central and marginal
rejection.

Using the parameter $\kappa$ of $chi\left(\mathbf{p};\kappa\right)$, the
relative preference of $chi\left(\mathbf{p};\kappa\right)$ to rejection along
the margins or in the centre can be directly controlled. Large $\kappa$
produce a pooled $p$-value which is powerful at detecting evidence spread
among all tests, while small $\kappa$ favour the detection of concentrated
evidence in a single test with extremes giving the widely-used
$Tip(\mathbf{p})$ and $Sto(\mathbf{p})$. The parameter $\kappa$ orders pooled
$p$-values of the $chi\left(\mathbf{p};\kappa\right)$ family by relative
centrality, simplifying the choice of pooled $p$-value and communication of
results. Finally, as it is based on Equation (4), it is an exact quantile-
based method which does not rely on asymptotic behaviour and which could,
hypothetically, be computed by hand with the aid of $\chi^{2}$ quantile
tables.

###  6.2 Comparing the chi-squared pooled $p$-value to the UMP benchmark

Recall the simulation studies that motivated the exploration of central and
marginal rejection levels. After a benchmark power computation, the power of
$HR\left(\mathbf{p};w\right)$ for $\alpha=0.05$ was evaluated under $H_{4}$
for a range of beta alternatives ($f=Beta(a,1/\omega+a(1-1/\omega))$) with KL
divergences from uniform ($D(a,\omega)$) spanning $e^{-5}$ to $e^{5}$. Correct
specification of $w$ was important: the larger the magnitude of $w-\omega$,
the larger the decrease in power of $HR\left(\mathbf{p};w\right)$ from
$HR\left(\mathbf{p};\omega\right)$. Under $H_{3}$, mis-specification did not
matter at all, the power of $HR\left(\mathbf{p};w\right)$ was dictated by the
the proportion of false null hypotheses ($\eta$) and the strength of evidence
against $H_{0}$ in each non-null hypothesis ($D(a,\omega)$). The parameter $w$
tunes $HR\left(\mathbf{p};w\right)$ to favour either weak evidence spread
among all tests, or strong evidence in only a few.

Finer selection of this tradeoff is achieved with the parameter $\kappa$ using
the $chi\left(\mathbf{p};\kappa\right)$ family of pooled $p$-values, but
controlling centrality is of little use if $chi\left(\mathbf{p};\kappa\right)$
is not powerful under the settings that motivated their definition. The power
of $chi\left(\mathbf{p};\kappa\right)$ at level $\alpha=0.05$ for each
$\kappa\in\\{e^{-8},e^{-4},1,2,e^{4},e^{8}\\}$ was therefore determined under
every setting from Section 4 using the same simulated samples generated under
$H_{4}$. Prior to the simulation, it is expected is that large $\kappa$ will
be uniformly more powerful than small $\kappa$, as under $H_{4}$ evidence is
spread among all tests. The results confirmed this expectation: the most
powerful $chi\left(\mathbf{p};\kappa\right)$ for all settings under $H_{4}$
was $chi\left(\mathbf{p};e^{8}\right)\approx chi\left(\mathbf{p};2981\right)$.
It is compared to the both the UMP and mis-specified
$HR\left(\mathbf{p};w\right)$ in Figure 10 by adding a dark grey line to
Figure 4.

Figure 10: A comparison of $chi\left(\mathbf{p};2981\right)$ to the UMP and
mis-specified $HR\left(\mathbf{p};w\right)$ under $H_{4}$.
$chi\left(\mathbf{p};2981\right)$ nearly UMP power more consistently than any
$HR\left(\mathbf{p};w\right)$, and so is more robust to $f$.

$chi\left(\mathbf{p};2981\right)$ has higher power than most mis-specified
$HR\left(\mathbf{p};w\right)$ for all settings in this case and is close to
the UMP more consistently than any $HR\left(\mathbf{p};w\right)$. Only when
$w\approx\omega$ does $HR\left(\mathbf{p};w\right)$ beat
$chi\left(\mathbf{p};2981\right)$, and so it is less robust to mis-
specification of $f$ than $chi\left(\mathbf{p};2981\right)$. It may therefore
be advisable to use $chi\left(\mathbf{p};\kappa\right)$ with a large $\kappa$
(or simply $Sto(\mathbf{p})$) when testing $H_{4}$ with beta alternatives in
the case where $\omega$ is not known, rather than risk the penalty of choosing
$w$ wrong when using $HR\left(\mathbf{p};w\right)$. This is despite the fact
that $HR\left(\mathbf{p};\omega\right)$ is UMP for this setting.

For the case where $\mathbf{p}$ was generated under $H_{3}$,
$chi\left(\mathbf{p};\kappa\right)$ was again computed for each
$\kappa\in\\{e^{-8},e^{-4},1,2,e^{4},e^{8}\\}$ over the 10,000 independent
samples for each setting of $D(a,\omega)$, $\omega$, and $\eta$ with $M=10$
from Section 4. Contour plots analogous to Figure 7 showing the differences in
power between $chi\left(\mathbf{p};\kappa\right)$ and
$HR\left(\mathbf{p};1\right)=chi\left(\mathbf{p};2\right)$ were generated. The
reference $HR\left(\mathbf{p};1\right)$ was chosen because it is a test shared
by both the $chi\left(\mathbf{p};\kappa\right)$ and
$HR\left(\mathbf{p};w\right)$ families.

|
---
(a)
|
---
(b)
Figure 11: Contours for the power of
$HR\left(\mathbf{p};1\right)=chi\left(\mathbf{p};2\right)$ minus (a)
$chi\left(\mathbf{p};e^{8}\right)\approx chi\left(\mathbf{p};2981\right)$ and
(b) $chi\left(\mathbf{p};e^{-8}\right)\approx
chi\left(\mathbf{p};0.003\right)$ by $\eta$ and $D(a,\omega)$ facetted by
$\omega$. Compared to Figure 7, (a) displays less of a penalty for the case of
concentrated evidence while still outperforming $HR\left(\mathbf{p};1\right)$
for evidence spread among all tests.

The patterns of power for $chi\left(\mathbf{p};\kappa\right)$ mimic those of
$HR\left(\mathbf{p};w\right)$: large $\kappa$ favour evidence spread among all
tests as do small $w$ in $HR\left(\mathbf{p};w\right)$. Despite this
similarity, $chi\left(\mathbf{p};2981\right)$ has higher power when applied to
the case of concentrated evidence and so is more robust under $H_{3}$. This is
seen clearly in a comparison of the bottom right corner of Figure 7 to Figure
11(a), the former shows a much larger and darker red region than the latter.

The $chi\left(\mathbf{p};\kappa\right)$ family also extends the range of
possible centrality parameters compared to $HR\left(\mathbf{p};w\right)$. As
$HR\left(\mathbf{p};1\right)=chi\left(\mathbf{p};2\right)$ is one of the
boundaries of the $w$ parameter range, no comparable pooled $p$-values to
$chi\left(\mathbf{p};\kappa\right)$ for $\kappa<2$ exist in the
$HR\left(\mathbf{p};w\right)$ family. Using
$chi\left(\mathbf{p};\kappa\right)$ therefore gives greater control over the
balance of central and marginal rejection than $HR\left(\mathbf{p};w\right)$,
though it seems exceptionally small $\kappa$ in
$chi\left(\mathbf{p};\kappa\right)$, or equivalently $Tip(\mathbf{p})$, should
only be used sparingly. Figure 11(b) shows that
$chi\left(\mathbf{p};0.003\right)$ loses power almost everywhere compared to
$chi\left(\mathbf{p};2\right)$ in exchange for higher power only in the case
of extreme evidence in a single test. Under $H_{3}$, very small values of
$\kappa$ should probably only be used if such a pattern of evidence is
strongly suspected.

The $chi\left(\mathbf{p};\kappa\right)$ family is therefore of interest both
practically and theoretically. It provides control over central and marginal
rejection under $H_{3}$ and robustly gives nearly UMP power for large values
of $\kappa$ under $H_{4}$. It has interpretable endpoints which cover a
greater range of centrality quotients than $HR\left(\mathbf{p};w\right)$ and
gives a means of controlling the bias towards central rejection present in all
quantile pooled $p$-values as $M$ increases.
$chi\left(\mathbf{p};\kappa\right)$ is a pooled $p$-value with great potential
as a practical tool for controlling the FWER when testing $H_{0}$.

##  7 Identifying plausible alternative hypotheses and selecting tests

The link between $\kappa$, the centrality quotient, and relative power in
regions of the $D(a,w),\eta$ plane under $H_{3}$ can be exploited to identify
alternatives to $H_{0}$ that could have plausibly generated $\mathbf{p}$.
Rather than selecting a particular $\kappa$ value, can consider all possible
$\kappa$ values simultaneously, compute $chi\left(\mathbf{p};\kappa\right)$
for each, and record

$\kappa_{\min}=\operatorname*{arg\,min}_{\kappa\in[0,\infty)}chi\left(\mathbf{p};\kappa\right)$
(23)

As each $\kappa$ value is associated with a particular centrality quotient,
each $\kappa$ identifies a particular region of relative power against others
in the $D(a,w),\eta$ plane under $H_{3}$. At the same time, $\kappa_{\min}$
reports the value of $\kappa$ which produces the smallest pooled $p$-value for
$\mathbf{p}$ and therefore suggests the $\kappa$ value where evidence against
$H_{0}$ is the strongest relative to other $\kappa$ values. As stronger
evidence leads to more frequent rejection and higher power when $H_{0}$ is
false, $\kappa_{\min}$ therefore links the evidence present in $\mathbf{p}$ to
a region in $D(a,w),\eta$ if we assume $H_{3}$ is truly used to generate the
data with $f=Beta(a,1/w+a(1-1/w))$.

###  7.1 Non-increasing beta densities

|
---|---
(a) | (b)
Figure 12: The (a) density and (b) central quantiles and median by $\kappa$
for the null case. All lines are flat and above the null quantiles from the
larger simulation as expected.

Begin with a demonstration of the sweep of $\kappa$ values for previously
explored cases by generating curves of $chi\left(;\kappa\right)$ by $\kappa$
for different densities under $H_{4}$. In each of the following, samples of
100 i.i.d. $p$-values from different beta distributions are generated
independently 1,000 times and $chi\left(;\kappa\right)$ is computed for a
sequence of $\kappa$ values chosen uniformly on the log scale. Let the
$i^{\text{th}}$ sample be $\mathbf{p}_{i}$ and the pooled $p$-value computed
using parameter $\kappa_{j}$ for $\mathbf{p}_{i}$ be
$\chi_{ij}=chi\left(\mathbf{p}_{i};\kappa_{j}\right)$. To provide context to
$\chi_{ij}$, a larger simulation of 100,000 samples was generated under
$H_{0}$ and the minimum of $chi\left(\mathbf{p}_{i};\kappa_{j}\right)$ for the
same sequence of $\kappa_{j}$ values was recorded. Figure 12(b) displays the
median and 0.5, 0.95, and 0.99 central quantiles for $f=Beta(1,1)$ (equivalent
to the null case) alongside the $Beta(1,1)$ density in 12(a). For reference
three dashed red lines at the observed 0.05, 0.01, and 0.001 quantiles of the
minimum pooled $p$-value over the 100,000 simulated null cases have been
added.

This case shows a flat median curve and flat central quantiles which are all
slightly above the corresponding minimum quantiles. The null case performs as
expected, $\kappa_{\min}$ would is distributed uniformly over the range of
$\kappa$ values and would produce values below the null quantiles at the
expected proportions. A contrasting case in shown in Figure 13, which uses the
same layout for an identical simulation carried out when $f=Beta(0.5,1)$.

|
---|---
a) | b)
Figure 13: The (a) density and (b) central quantiles and median by $\kappa$
for $p$-values generated identically and independently from a Beta(0.5, 1)
distribution. The minimum around $\kappa=2$ corresponds to the UMP.

Displaying the median and the same central quantiles as before, there is a
unique minimum at $\kappa=2$, a lower right end to the curve than the left
end, and a generally lower value across its entire length. If one of these
curves was observed in practice, $\kappa_{\min}\approx 2$ would be chosen and
larger $\kappa$ values may not be fully ruled out. This conclusion would be
correct: under $H_{4}$ with $f=Beta(0.5,1)$ the UMP pooled $p$-value is
$HR\left(\mathbf{p};w\right)$ with $w=(1-a)/(b-a)=1$ and
$HR\left(\mathbf{p};1\right)=chi\left(\mathbf{p};2\right)=Fis(\mathbf{p})$.
Furthermore, the power investigations in Section 6.2 demonstrate that
$chi\left(\mathbf{p};2981\right)\approx Sto(\mathbf{p})$ is nearly as powerful
as the UMP for all $w$ under $H_{4}$. This confirms empirically that the level
of the curve of $chi\left(\mathbf{p};\kappa\right)$ over $\kappa$ corresponds
to the relative power of pooled $p$-values in
$chi\left(\mathbf{p};\kappa\right)$ for this beta distribution.

Of course, this conclusion should be expanded to $H_{3}$ and so mixture of
$Beta(0.1,1)$ and $Beta(1,1)$ distributions is considered. The first
distribution provides strong evidence against the null hypothesis while the
second corresponds to the uniform distribution, and so contains no evidence
against the null. Mixing these such that the probability of drawing from
$Beta(0.1,1)$ is 0.05 and the probability of drawing from the null is 0.95 we
are placed in the $D(a,w),\eta$ space at $0.3,0.05$. Simulating this as for
the null and $Beta(0.5,1)$ cases and displaying the central quantiles of
$\chi_{ij}$ alongside the mixture density gives Figure 14.

|
---|---
(a) | (b)
Figure 14: The (a) density and (b) central quantiles and median by $\kappa$
for the mixture 0.05Beta(0.1, 1) + 0.95Beta(1, 1). Small $\kappa$ values
provide the smallest pooled $p$-values, and hence power at detecting this
alternative.

The central quantiles are more variable for this case than the unmixed
densities because the probability of generating any $p$-values from
$Beta(0.1,1)$ is small and so many samples would have included only null
$p$-values. Nonetheless, the median has a unique minimum near $\kappa=0.01$,
and is generally lower for small $\kappa$ than large $\kappa$. Considering the
coordinates of this case in the $D(a,w),\eta$ plane, this is completely
consistent with the earlier investigations of power where small $\kappa$
values were most powerful for strong evidence concentrated in a few tests. In
practice, seeing the median curve would cause us to suspect this case
correctly.

###  7.2 Identifying a region of alternative hypotheses

While these one-to-one comparisons between densities and $\kappa$ curves help
to demonstrate the link between $\kappa_{\min}$ and the alternative densities
used to generate $\mathbf{p}$, they are not incredibly informative. Given
$\mathbf{p}$ and supposing we generate such a curve of
$chi\left(\mathbf{p};\kappa\right)$ by $\kappa$, we would need to sort through
an incredible number of density-curve pairs to identify plausible alternatives
corresponding to the curve obtained.

Instead, consider a more automated approach. Given a collection of $p$-values,
this generates a curve by sweeping parameter values of $\kappa$, identifies
the minimum $\kappa$ values (or any below a particular threshold), and maps
these back onto the plane of strength and prevalence depicted in, for example,
Figure 11. This requires a detailed guide of where each $\kappa$ value is most
powerful in the $\eta,D(a,w)$ plane so that $\kappa_{\min}$ or the range of
significant $\kappa$ values can be placed accurately. Therefore, a simulation
was carried out over 20 $\ln(w)$ values evenly spaced from $-6$ to $0$, $\eta$
values from 0 to 1 in increments of 1/80, and 80 $\ln D(a,w)$ values evenly
spaced between $-5$ and $5$. For each combination, 10,000 samples of 80
$p$-values were generated with $80\eta$ following the beta distribution
specified by $\ln(w)$ and $\ln D(a,w)$ and $80(1-\eta)$ following the uniform
distribution.111111This resolution was not the only one tried, similar
experiments were carried out for $M=10$, $20,$ and $40$ and the only impact of
increasing $M$, the number of steps in $\eta$, and the number of steps in $\ln
D(a,w)$ was increasing resolution of the same patterns. This suggests that
these patterns do not depend on the sample size.

Each of the 10,000 samples then had pooled $p$-values computed over a sweep of
65 $\ln\kappa_{i}$ values evenly spaced from $-8$ to $8$ and the power at
level $\alpha=0.05$ was computed for the rejection rule
$chi\left(\mathbf{p};\kappa_{i}\right)\leq 0.05$. The $\kappa_{i}$ with the
greatest power for each combination corresponds to $\kappa_{\min}$ for that
combination because rejection is determined by thresholding the pooled
$p$-value and so a higher power implies a lower distribution of the pooled
$p$-value at a given point. This distribution with a lower location will
result in an equal or lower quantile curve to all others. Bivariate
discretized Gaussian smoothing is applied to each
$chi\left(\mathbf{p};\kappa_{i}\right)$ power surface in $\eta,D(a,w)$ for
each $w$ value in order to obtain a smoothed estimate of the power surface
minimally impacted by random binomial noise. This was completed only because
none of the investigations carried out indicated discontinuities in the power
or distribution of $p$-values by $\kappa_{i}$.

For each $w$, the power surfaces of every
$chi\left(\mathbf{p};\kappa_{i}\right)$ in $\eta,D(a,w)$ were then compared to
the maximum among them point-wise. This is motivated by the simpler case shown
in Figure 13, as several $\kappa_{i}$ values are often equally powerful for a
given setting. Specifically, the comparison was a binomial test of the
difference in proportions using a normal approximation at 95% confidence. A
surface was deemed equal to the maximum power at that point if the test failed
to reject the null hypothesis of equal proportions.121212For powers $p_{1}$
and $p_{2}$ computed over the same number of trials $n$, this computes
$z=\frac{\sqrt{n}(p_{1}-p_{2})}{\sqrt{2p(1-p)}}$ where
$p=\frac{p_{1}+p_{2}}{2}$ and then compares $z$ to normal critical values. By
the CLT, $z\sim N(0,1)$ approximately for large $n$, and as 10,000 simulations
are performed for each power estimate, this approximation should be quite
accurate. For each $\kappa$ and $w$, all of this pre-processing gave a matrix
in $\eta$ and $\ln D(a,w)$ indicating whether
$chi\left(\mathbf{p};\kappa_{i}\right)$ achieved the maximum power for that
combination for every $\kappa_{i}$. To produce a final summary in $\eta$ and
$\ln D(a,w)$ alone, these indicators were summed over $w$ for each
$\kappa_{i}$. Finally, masks were added in the top right and bottom left
corners where all methods are equally powerful with powers 1 and 0.05
respectively to make the meaningful patterns more visible. Figures 15(a) - (d)
display these sums (counts of cases in $w$ where $\kappa_{i}$ achieved maximum
power) for several $\kappa_{i}$ in a given $\eta,\ln D(a,w)$ region, with
guide histograms on each row and column added to quickly indicate the relative
marginal frequencies. Each plot is therefore rich with both marginal and joint
information on the regions where a particular $\kappa_{i}$ is most powerful.

|
---|---
a) $\ln\kappa=8$ | b) $\kappa=2$
|
c) $\ln\kappa=-1$ | d) $\ln\kappa=-4$
Figure 15: Likely alternatives for a range of $\kappa$ values. For full
coverage of $D(a,w)$, $w$ was chosen uniformly on a log scale.

Consistent with previous investigations, this map shows that the regions where
the small $\kappa$ values are most powerful correspond to small $\eta$ values.
The mode of the histogram of $\eta$ values increases steadily in $\kappa$
until it is near one when $\kappa=e^{8}$. For $\kappa=e^{-4}$, the pooled
$p$-value is only most powerful for settings with $\eta<0.1$, such that a
minimum of the parameter curve below $e^{-4}\approx 0.02$ indicates a small
minority of tests are significant.

Given that these plots display counts of cases where a particular
$chi\left(\mathbf{p};\kappa\right)$ is most powerful, choosing to select $w$
evenly-spaced on the log scale inadvertently places greater weight on small
values of $w$ and under-samples large values in order to achieve more complete
coverage of $D(a,w)$. Even for moderate $w$ floating point representation
limits prevent the computation of $a$ and $b$ for the beta distributions with
large KL divergences. This complete coverage of the strength of evidence is
one of two possible perspectives, with the other focused on even exploration
of the parameter space. For this parameter-based perspective, the exact same
procedure was performed but with 20 $w$ values selected at even increments
from 0.05 to 1. Figures 16(a) - (d) present the same heatmaps as Figure 15
from this perspective.

|
---|---
a) $\ln\kappa=8$ | b) $\kappa=2$
|
c) $\ln\kappa=-1$ | d) $\ln\kappa=-4$
Figure 16: Likely alternatives for a range of $\kappa$ values. $w$ was chosen
uniformly for these images, leading to worse coverage of $D(a,w)$ but more
appropriate coverage of $w$.

There are some noteworthy differences between this and the first set of
heatmaps. The bias towards smaller proportions and stronger evidence in the
first is quite clear when it is compared to the second, which generally shows
similar shapes but more evenly distributed saturation across this shape. This
leads to changes in the regions suggested for a particular $\kappa_{\min}$,
but these are typically minor. The biggest difference occurs for large
$\kappa$, where the bias towards small values in Figure 15 obscures all of the
internal variation in the middle top that can be seen in Figure 16.

Without these plots, an analyst would be left trying to identify alternatives
from a density estimate. Besides showing comparable information about the
prevalence of evidence to a density estimate in the histogram along the right
side of the plot, these plots of plausible alternatives give information about
likely strengths of evidence and regions for the combination of both. By
leveraging the links between the centrality quotient, $\kappa$, and the
distribution and strength of evidence in $\mathbf{p}$, these maps provide
richer and clearer information.

###  7.3 Selecting a subset of tests

Perhaps the most important part of the alternative heatmaps presented in
Figures 15 and 16 are the histograms along the right margin that indicate the
likely prevalence of evidence in the data. Once $\kappa_{\min}$ has been
determined using a sweep of $\kappa$ values, and a plausible set of
alternatives has been identified using these alternative heatmaps, the
corresponding range of proportions can be used to identify a subset of tests
of interest. If false positives are less problematic to analysis than false
negatives, the upper bound of this range might be taken, with the other bound
taken if the opposite is true. In either case, suppose the chosen proportion
is $\eta^{*}$, then the $M\eta^{*}$ largest values of
$F^{-1}(1-p_{i};\kappa_{\min})$ are the tests most contributing to the small
value of $chi\left(\mathbf{p};\kappa_{\min}\right)$ and so are the tests of
greatest interest that can be selected for further investigation.

###  7.4 Centrality in other beta densities

Until now, it was always assumed that $p$-values follow a non-increasing beta
density when the null hypothesis is false. This is a reasonable assumption,
many statistical tests have this property for the rejection rule thresholding
the $p$-value at $\alpha$. Relaxing this assumption, however, allows an
exploration into how $chi\left(\mathbf{p};\kappa\right)$ behaves for a broader
variety of densities and whether centrality is still a useful concept under
these other distributions of non-null $p$-values.

First, consider the case of a strictly increasing density under $H_{4}$.
Whether or not this case is interesting is a matter of opinion as under the
convention that small $p$-values are evidence against $H_{0}$ such a density
produces even less evidence against $H_{0}$ than the null distribution itself.
It would be reasonable to expect, then, that this case produces only very
large $chi\left(\mathbf{p};\kappa\right)$ for all $\kappa$ values. Following
the same procedure as Section 7.1, this expectation is tested for
$Beta(1,0.5)$, resulting in the curve and density displayed in Figure 17.

|
---|---
(a) | (b)
Figure 17: The (a) density and (b) central quantiles and median by $\kappa$
for $p$-values from a Beta(1, 0.5) distribution. The pooled $p$-value is
generally large compared to the empirical curve minimum quantiles.

As expected, this setting gives only large $chi\left(\mathbf{p};\kappa\right)$
values for every $\kappa$. There is a slight dip to small $p$-values for small
$\kappa$, likely a result of the small $p$-values that still occur for this
density occasionally, but it barely crosses the null reference lines. Perhaps
a more realistic case is a density that rarely, if ever, produces minimum
$p$-values small enough to warrant rejection alone, but does tend to produce
far more $p$-values less than 0.5 than expected under the null hypothesis. For
such a setting, the previous investigations into centrality suggest that
$\kappa_{\min}$ should be large. An example is $f=Beta(2,4)$ under $H_{4}$,
shown in Figure 18.

|
---|---
(a) | (b)
Figure 18: The (a) density and (b) central quantiles and median of a
$p$-values following a Beta(2, 4) distribution. The absence of very small
$p$-values and bias towards smaller ones means large $\kappa$ values are most
powerful.

Once again, this plot matches the expectation reasoned from centrality,
despite the relaxation of the assumptions used to motivate centrality. Both of
these examples suggest that the concepts of central and marginal rejection may
have use beyond non-increasing densities, and provide a promising framework
for future investigation.

##  8 The PoolBal package

There is no lack of packages available to pool $p$-values in R. The most
recent of these, poolr (Cinar and Viechtbauer, 2021), lists 8 others all
providing piecemeal coverage of every pooled $p$-value proposal131313These are
Dewey (2022) Zhang et al. (2020); Wilson (2019); Yi and Pachter (2018); Poole
and Gibbs (2015); Dai et al. (2014); Schröder et al. (2011); Zhao (2008). Most
of these packages cover a subset of pooling functions or implement adjustments
for dependence rather than attempting to be the complete package for pooling
$p$-values.. Rather than re-implement the functionality provided by these
packages, the PoolBal package aims primarily to support the evaluation of the
central rejection level, marginal rejection level, and centrality quotient for
these and any future packages which pool $p$-values. As they both allow some
tuning of centrality, these core functions are supported by implementations of
$chi\left(\mathbf{p};\kappa\right)$ and $HR\left(\mathbf{p};\kappa\right)$
along with functions to evaluate the Kullback-Leibler divergence for general
densities and compute it for the beta density in particular. This is meant to
make the adoption of the framework provided in this work as simple as
possible.

Briefly summarized, the functionality of PoolBal includes

klDiv, betaDiv:

compute the Kullback-Leibler divergence for arbitrary densities and the
uniform to Beta case, respectively

findA:

invert a given Kullback-Leibler divergence and most powerful test parameter
$w$ to identify the unique Beta parameter $a$ that corresponds to this setting

pBetaH4, pBetaH3:

helpers to generate $\mathbf{p}$ under under $H_{4}$ and $H_{3}$

estimatePc, estimatePrb, estimateQ:

wrappers for uniroot from base that estimate the central rejection level,
marginal rejection level at $b$, and centrality quotient for an arbitrary
function

chiPool:

an implementation of $chi\left(\mathbf{p};\kappa\right)$

chiPc, chiPr, chiQ:

functions to compute the central rejection level, marginal rejection level,
and centrality quotient of $chi\left(\mathbf{p};\kappa\right)$ using Equations
(20), (21), and (22)

chiKappa:

a wrapper for uniroot from base that inverts a given centrality quotient to
give the $\kappa$ value in $chi\left(\mathbf{p};\kappa\right)$ with the
corresponding quotient

hrStat, hrPool:

compute $l_{HR}(\mathbf{p};w)$ and $HR\left(\mathbf{p};w\right)$ for
$\mathbf{p}$, with the $p$-value determined empirically using simulated null
data

hrPc, hrPr, hrQ:

functions to compute the central rejection level, marginal rejection level,
and centrality quotient of $HR\left(\mathbf{p};\kappa\right)$ using simulation
and uniroot

altFrequencyMat:

function allowing access to a summarized version of the simulation results
from Section 7.2

marHistHeatMap:

function which generates heatmaps with marginal histograms, that is
visualizations such as those in Figure 16.

The package can be found on the author’s GitHub.

##  9 Conclusion

When presented with $M$ $p$-values from independent tests of hypotheses
$H_{01},\dots,H_{0M}$, a natural way to control the family-wise error rate
(FWER) is by pooling these $p$-values using a function $g(\mathbf{p})$. If
$g(\mathbf{p})$ is constructed using the sum of quantile transformations or
the order statistics of the $p$-values, then the rejection rule
$g(\mathbf{p})\leq\alpha$ controls the FWER at $\alpha$. Selecting between the
many possible $g(\mathbf{p})$ requires the choice of an alternative hypothesis
from the telescoping set $H_{1}\supset H_{2}\supset H_{3}\supset H_{4}$ in
order to determine their powers against these alternatives. $H_{3}$ and
$H_{4}$, though the most restrictive, still require the choice of $\eta$, the
prevalence of non-null $p$-values in $\mathbf{p}$, and $f$, the distribution
of these non-null values. An obvious choice for $f$ is the beta distribution
restricted to be non-decreasing, as this biases non-null $p$-values lower than
null $p$-values. By using $\eta$ and the Kullback-Leibler divergence of $f$
from the uniform distribution, both the prevalence and strength of non-null
evidence can be measured.

If all the evidence is non-null, i.e. $\eta=1$, the pooled $p$-value based on
$l_{w}(\mathbf{p})=w\sum_{i=1}^{M}\ln p_{i}-(1-w)\sum_{i=1}^{M}\ln(1-p_{i})$
is uniformly most powerful (UMP) but is sensitive to the specification of its
parameter $w\in[0,1]$. Incorrectly choosing this parameter, i.e. selecting a
value that does not match the true generative distribution, costs power to
reject the alternative hypothesis $H_{4}$. When $\eta\neq 1$, both the
prevalence and strength of non-null evidence dictate the most powerful choice
of $w$. Small values of $w$ are more powerful for weak evidence spread among
all tests while large values are better at detecting strong evidence in a few
tests.

This reflects a more universal pattern in pooled $p$-values and motivates a
new paradigm for selecting and analyzing them. The marginal level of rejection
at $\alpha$, the largest individual $p$-value that leads to rejection at
$\alpha$ when all other $p$-values are 1, and the central rejection level at
$\alpha$, the largest value simultaneously taken by all elements of
$\mathbf{p}$ which still leads to rejection at $\alpha$, characterize this
paradigm. By defining the central and marginal rejection level, a number of
fundamental properties can be proven. Among them, the central rejection level
of a pooled $p$-value satisfying some mild conditions is always greater than
or equal to the marginal rejection level, with equality occurring only for
$Tip(\mathbf{p})$. This order allows a centrality quotient to be defined which
summarizes the preference of a pooled $p$-value to diffuse or concentrated
evidence with a value in $[0,1]$.

In order to control this quotient, a pooled $p$-value based on
$\chi^{2}_{\kappa}$ quantile transformations was defined,
$chi\left(\mathbf{p};\kappa\right)$. By choosing the degrees of freedom
$\kappa\in[0,\infty)$, arbitrary control over the centrality of the pooled
$p$-value is obtained. Increasing $\kappa$ raises the centrality quotient, and
decreasing it drops the quotient. Furthermore, the limiting cases of
$\kappa=0$ and $\kappa\rightarrow\infty$ correspond to $Tip(\mathbf{p})$, the
minimum order statistic $p$-value, and $Sto(\mathbf{p})$, the normal quantile
transformation $p$-value. Both of these limiting pooled $p$-values are classic
pooling functions which have been used and studied widely in the literature.
$chi\left(\mathbf{p};\kappa\right)$ therefore provides a means to balance an
important aspect of pooling $p$-values with a single parameter that has ready
interpretation along its range. Comparing its power to
$HR\left(\mathbf{p};w\right)$ under $H_{3}$ and $H_{4}$,
$chi\left(\mathbf{p};\kappa\right)$ loses less power than
$HR\left(\mathbf{p};w\right)$ with $w$ mis-specified.
$chi\left(\mathbf{p};\kappa\right)$ is therefore more robust, and demonstrates
that the central and marginal rejection paradigm is instructive to predict
which version of $chi\left(\mathbf{p};\kappa\right)$ will be most powerful for
a particular alternative hypothesis. $chi\left(\mathbf{p};\kappa\right)$ and
the centrality quotient are both potent tools for pooling $p$-values to
control the FWER.

## References

  * Birnbaum (1954) Allan Birnbaum.  Combining independent tests of significance.  _Journal of the American Statistical Association_ , 49(267):559–574, 1954.
  * Canal (2005) Luisa Canal.  A normal approximation for the chi-square distribution.  _Computational Statistics & Data Analysis_, 48(4):803–808, 2005.
  * Cinar and Viechtbauer (2021) Ozan Cinar and Wolfgang Viechtbauer.  _poolr: Methods for Pooling P-Values from (Dependent) Tests_ , 2021\.  URL https://CRAN.R-project.org/package=poolr.  R package version 1.0-0.
  * Cinar and Viechtbauer (2022) Ozan Cinar and Wolfgang Viechtbauer.  The poolr package for combining independent and dependent p values.  _Journal of Statistical Software_ , 101:1–42, 2022.
  * Dai et al. (2014) Hongying Dai, J Steven Leeder, and Yuehua Cui.  A modified generalized Fisher method for combining probabilities from dependent tests.  _Frontiers in Genetics_ , 5:32, 2014.
  * Dewey (2022) Michael Dewey.  metap: Meta-analysis of significance values. r package version 1.8, 2022\.
  * Edgington (1972) Eugene S. Edgington.  An additive method for combining probability values from independent experiments.  _The Journal of Psychology_ , 80(2):351–363, 1972\.
  * Fisher (1932) Ronald Aylmer Fisher.  _Statistical Methods for Research Workers_.  Oliver and Boyd, Edinburgh, 1932.
  * Goutis et al. (1996) Constantinos Goutis, George Casella, and Martin T. Wells.  Assessing evidence in multiple hypotheses.  _Journal of the American Statistical Association_ , 91(435):1268–1277, 1996.
  * Hawkins and Wixley (1986) Douglas M. Hawkins and R.A.J. Wixley.  A note on the transformation of chi-squared variables to normality.  _The American Statistician_ , 40(4):296–298, 1986\.
  * Heard and Rubin-Delanchy (2018) Nicholas A. Heard and Patrick Rubin-Delanchy.  Choosing between methods of combining $p$-values.  _Biometrika_ , 105(1):239–246, 2018.
  * Inglot (2010) Tadeusz Inglot.  Inequalities for quantiles of the chi-square distribution.  _Probability and Mathematical Statistics_ , 30(2):339–351, 2010.
  * Joyce (2011) James M Joyce.  Kullback-Leibler divergence.  In _International Encyclopedia of Statistical Science_ , pages 720–722. Springer, 2011.
  * Kocak (2017) Mehmet Kocak.  Meta-analysis of univariate p-values.  _Communications in Statistics – Simulation and Computation_ , 46(2):1257–1265, 2017.
  * Koziol and Perlman (1978) James A. Koziol and Michael D. Perlman.  Combining independent chi-squared tests.  _Journal of the American Statistical Association_ , 73(364):753–763, 1978.
  * Lancaster (1961) H.O. Lancaster.  The combination of probabilities: an application of orthonormal functions.  _Australian Journal of Statistics_ , 3(1):20–33, 1961.
  * Littell and Folks (1971) Ramon C. Littell and J. Leroy Folks.  Asymptotic optimality of Fisher’s method of combining independent tests.  _Journal of the American Statistical Association_ , 66(336):802–806, 1971.
  * Loughin (2004) Thomas M. Loughin.  A systematic comparison of methods for combining p-values from independent tests.  _Computational Statistics & Data Analysis_, 47(3):467–485, 2004.
  * Mudholkar and George (1977) Govind S. Mudholkar and E.O. George.  The logit statistic for combining probabilities – an overview.  Technical report, Rochester University Department of Statistics, 1977\.
  * Owen (2009) Art B Owen.  Karl Pearson’s meta-analysis revisited.  _The Annals of Statistics_ , 37(6B):3867–3892, 2009.
  * Pearson (1933) Karl Pearson.  On a method of determining whether a sample of size $n$ supposed to have been drawn from a parent population having a known probability integral has probably been drawn at random.  _Biometrika_ , 25(3/4):379–410, 1933.
  * Poole and Gibbs (2015) William Poole and David Gibbs.  _EmpiricalBrownsMethod: Uses Brown’s method to combine p-values from dependent tests_ , 2015.  URL https://github.com/IlyaLab/CombiningDependentPvaluesUsingEBM.git.  R package version 1.28.0.
  * Schröder et al. (2011) Markus S Schröder, Aedín C Culhane, John Quackenbush, and Benjamin Haibe-Kains.  survcomp: an r/bioconductor package for performance assessment and comparison of survival models.  _Bioinformatics_ , 27(22):3206–3208, 2011.
  * Siegel (1979) Andrew F. Siegel.  The noncentral chi-squared distribution with zero degrees of freedom and testing for uniformity.  _Biometrika_ , 66(2):381–386, 1979.
  * Stouffer et al. (1949) Samuel A. Stouffer, Edward A. Suchman, Leland C. DeVinney, Shirley A. Star, and Robin M. Williams Jr.  _The American soldier: Adjustment during army life. (Studies in social psychology in World War II), vol. 1_.  Princeton University Press, 1949.
  * Tippett (1931) Leonard Henry Caleb Tippett.  _The Methods of Statistics_.  London: Williams & Norgate Ltd., 1931.
  * Wallis (1942) W. Allen Wallis.  Compounding probabilities from independent significance tests.  _Econometrica, Journal of the Econometric Society_ , 10(3/4):229–248, 1942.
  * Westberg (1985) Margareta Westberg.  Combining independent statistical tests.  _Journal of the Royal Statistical Society: Series D (The Statistician)_ , 34(3):287–296, 1985.
  * Wilkinson (1951) Bryan Wilkinson.  A statistical consideration in psychological research.  _Psychological Bulletin_ , 48(2):156, 1951.
  * Wilson (2019) Daniel J. Wilson.  The harmonic mean p-value for combining dependent tests.  _Proceedings of the National Academy of Sciences_ , 116(4):1195–1200, 2019.
  * Yi and Pachter (2018) Lynn Yi and Lior Pachter.  _aggregation: p-Value Aggregation Methods_ , 2018.  URL https://cran.r-project.org/package=aggregation.  R package version 1.0.1.
  * Zaykin et al. (2007) Dmitri V. Zaykin, Lev A. Zhivotovsky, Wendy Czika, Susan Shao, and Russell D. Wolfinger.  Combining p-values in large-scale genomics experiments.  _Pharmaceutical Statistics_ , 6(3):217–226, 2007\.
  * Zhang et al. (2020) Hong Zhang, Tiejun Tong, John Landers, and Zheyang Wu.  Tfisher: a powerful truncation and weighting procedure for combining p-values.  _Annals of Applied Statistics_ , 14(1):178–201, 2020.
  * Zhao (2008) Jing Hua Zhao.  gap: Genetic analysis package.  _Journal of Statistical Software_ , 23:1–18, 2008.