# Synthetic Power Analyses: Empirical Evaluation and Application to Cognitive
Neuroimaging

Peiye Zhuang
Dept. of Comupter Science,
University of Illinois
at Urbana-Champaign
<EMAIL_ADDRESS>Bliss Chapman  Apple
<EMAIL_ADDRESS>Ran Li  Google
<EMAIL_ADDRESS>Sanmi Koyejo  Dept. of Comupter Science
& Beckman Institute,
University of Illinois
at Urbana-Champaign
<EMAIL_ADDRESS>
###### Abstract

In the experimental sciences, statistical power analyses are often used before
data collection to determine the required sample size. However, traditional
power analyses can be costly when data are difficult or expensive to collect.
We propose synthetic power analyses; a framework for estimating statistical
power at various sample sizes, and empirically explore the performance of
synthetic power analysis for sample size selection in cognitive neuroscience
experiments. To this end, brain imaging data is synthesized using an implicit
generative model conditioned on observed cognitive processes. Further, we
propose a simple procedure to modify the statistical tests which result in
conservative statistics. Our empirical results suggest that synthetic power
analysis could be a low-cost alternative to pilot data collection when the
proposed experiments share cognitive processes with previously conducted
experiments.

###### Index Terms:

fMRI, GANs, power analyses

##  I Introduction

Cognitive neuroscience studies how the brain produces intelligent behavior.
One of the most common measurement tools for brain activity is functional
magnetic resonance imaging (fMRI), used to gain insight into how patterns in
brain activity correspond to cognitive functions. The use of imaging tools to
understand cognition is known as cognitive neuroimaging. This paper proposes
generative modeling techniques that provide a promising methodological
approach to improve statistical analysis in cognitive neuroimaging.

Improvements to statistical analysis are particularly timely due to an
increasing focus on the quality of data-dependent analysis in science and
medicine, probably most publicly in the psychological sciences [1, 2] One of
the primary quantities of interest in statistical analyses of scientific data
is statistical power; the probability that a test rejects the null hypothesis
given that the alternative hypothesis is correct. Lower power reflects a low
probability that the test correctly detects a statistically significant
effect. Neuroscientists typically aim for 80% power [3]. However, in recent
years, the statistical power of published neuroscience studies has been
demonstrated to be much lower than initially claimed, especially in published
fMRI experiments [4].

Prospective power analyses promise an optimal experiment design that maximizes
statistical power and reduces the waste of resources on futile experiments or
additional subjects. In traditional power analysis, real data is collected and
used to determine the sample size needed to achieve the desired experimental
power. Unfortunately, an MRI machine with resolution at the level required for
neuroscience research costs between 0.5 and 3 million dollars [5], and the
scanning can cost about $500+ an hour [6]. Thus, resources are not always
available to run studies that could supply data for power analysis. This
barrier lowers the quality of proposed research experiments and slows overall
progress in neuroscience.

One approach to circumvent the data problem is to use stand-in data from other
research groups for power analysis. However, even when the borrowed data is a
good substitute for the proposed experiment, this approach requires planning
well in advance and slows turnaround times. More importantly, finding stand-in
data can be challenging. An alternative approach is to determine effect sizes
and variance estimates from previously published studies. Again, even when one
can find a sufficiently similar study, the experiment can suffer from
underpowered estimates because published articles are biased towards reporting
significant effects [1, 2].

We propose the use of modern generative neural network models to synthesize
diverse, high-quality brain images. Specifically, we use the three-dimensional
Improved Conditional Wasserstein Generative Adversarial Network (ICW-GANs)
model [7]. The observation motivates our use of this generative approach to
produce unlimited synthetic data, which, in turn, promises informed power
analyses with minimal cost overhead. In particular, our model can synthesize
cognitive processes corresponding to labels that are not present in the
original training set, thus enabling the direct application of synthetic fMRI
data to real-world power analyses for new experiments.

The primary aim of this manuscript is to empirically evaluate the performance
of synthetic power analyses through simulated data and neuroimaging
experiments. We compare the power analysis results of synthetic to real data
using both the classic two-sample t-test and the non-parametric maximum mean
discrepancy (MMD) test [8]. Our experiments demonstrate that the synthetic
data has similar distributional characteristics as real data, and that power
analysis results calculated with synthetic data are similar to those
calculated with real data. Taken together, our results suggest that synthetic
data can be a reliable and low-cost substitute for real data used in cognitive
neuroimaging experiment design.

##  II Related Work

Availability of brain imaging Data. The lack of sufficient brain imaging data
may stifle the progress in computational cognitive neuroimaging research [9].
This issue is more evident in decoding studies predicting cognitive and
behavioral outcomes of brain imaging experiments, particularly when one would
like to use machine learning methods [10]. As a result, there is a dearth of
research on sophisticated predictive models for high-dimensional brain images
[11, 12, 13].

Deep learning in Neuroscience. Neural networks have been used extensively for
classifying brain imaging data. For instance, deep networks have been used to
extract features of fMRI brain images to classify brain states [14, 15], and
2D and 3D neural networks to classify fMRI brain data [13]. fMRI data of video
stimuli have been decoded and classified data into visual categories [16].
Similarly, [13] extracted features from 4D fMRI data and used deep learning
methods for discrimination of cognitive processes.

Generative Adversarial Networks (GANs). Generative Adversarial Networks (GANs)
[17] are a promising recent development in generative modeling. GANs have
demonstrated a capacity to capture complex data distributions through a non-
cooperative two-player game formulation. In the GAN framework, a generator $G$
transforms samples from a simple distribution into complex high-dimensional
data. At the same time, a critic $D$ attempts to distinguish between the
synthetic and real data. Let $x$ represent the data with empirical data
sensity $p_{data}(x)$. The variable $z$ represents the low-dimensional latent
variable with a simple distribution $p(z)$ such as $\mathcal{N}(0,I)$. The
generator and the critic optimize value function $V(G,D)$ through a two-player
minimax game:

$\displaystyle\min_{G}\max_{D}V(D,G)=$ $\displaystyle\mathbb{E}_{x\sim
p_{data}(x)}[\log(D(x))]$ $\displaystyle+\mathbb{E}_{z\sim
p_{z}(z)}[\log(1-D(G(z)))].$

In the conditional GAN [18], the model is provided additional information at
training time about class labels or features. 3D-GANs [18] extend GANs to 3D
object generation through the use of three-dimensional convolutions. However,
the classical training procedure is brittle without other stabilization
techniques. The conditional information can be manipulated to allow for finer-
grained control over data generation. The Wasserstein GAN (WGAN) [19] uses
Wasserstein-1 distance as a loss metric that improves the stability of the
training procedure. Improved Wasserstein GAN (WGAN-gp) [20] further extends
these stability improvements by introducing a gradient penalty term in the
critic network that replaces weight clipping and enforces a Lipschitz
constraint. [7] first proposed an Improved Conditional Wasserstein Generative
Adversarial Networks (ICW-GAN) to the synthesis of fMRI brain imaging data.
The objective function of a 3D Conditional Wasserstein ICW-fMRI-GAN is as
follows:

$\displaystyle L=$ $\displaystyle\mathbb{E}_{\bm{z}\sim
P_{\bm{z}}(z)}[1-D(G(\bm{z}|\bm{y}))]$ (1)
$\displaystyle-\mathbb{E}_{\bm{x}\sim P_{data}(x)}[D(\bm{x}|\bm{y})]$
$\displaystyle+\lambda\mathbb{E}_{\hat{\bm{x}}\sim
P_{\hat{\bm{x}}}(\hat{x})}[(\left\lVert\nabla_{\hat{\bm{x}}}D(\hat{\bm{x}}|\bm{y})\right\rVert_{2}-1)^{2}],$

where $\bm{y}$ denotes the volume labels,
$\hat{\bm{x}}\leftarrow\epsilon\bm{x}+(1-\epsilon)G(\bm{z})$, $\epsilon\sim
U(0,1)$ and $\lambda$ is a gradient penalty coefficient.

Two-sample test. In statistical testing, a two-sample test is used to deduce
if two samples are generated from the same distribution or two separate
distributions. We consider two main types of two-sample tests, as outlined in
the following.

i) A parametric test based on explicit distributional assumptions e.g.,
t-test[21, 22, 23, 24] assumes that the two distributions are both Gaussian
distributions.

ii) A non-parametric test that employs weaker assumptions, while yielding
effective tests. One example is the maximum mean discrepancy (MMD) test [25,
26, 27]. Suppose there are two distributions ${\displaystyle P(X)}$ and
${\displaystyle Q(Y)}$, the MMD between them can be formulated as follows:

${\displaystyle{\text{MMD}}(P,Q)=\sup_{f\in\mathcal{F}}\left(\mathbb{E}_{X}[f(X)]-\mathbb{E}_{Y}[f(Y)]\right)},$

where $f$ is a function in a function set $\mathcal{F}$. Usually, a unit ball
in a reproducing kernel Hilbert space (RKHS)[28] is used as the MMD function
classes $\mathcal{F}$. Classical MMD methods compute $L_{2}$ distance between
two distributions in the kernel space. Recent work [29] has shown that
replacing the distance with the $L_{1}$ distance can improve test power.

##  III Methods

###  III-A ICW-GANs

We use an ICW-GAN, a class-conditional model proposed in [7] to synthesize
fMRI data conditioned on selected labels. The objective function of an ICW-GAN
is shown in Eq. 1. With regard to its architecture, the generator consists of
a four-layer fully-connected convolutional layers, batch normalization, ReLU
layers added in between, and a sigmoid layer at the end. We use 3-dimensional
convolutions which enable modeling of the spatial relations of high-
dimensional brain images. The critic has a similar architecture as the
generator except for the final layer that uses a linear activation. We train
for 15,000 iterations with batches of size 50, an input noise vector of length
128, and a gradient penalty $\lambda=1$. Both the critic and the generator are
optimized with the RMSprop optimizer [30] with a learning rate of $1e-3$ and
are updated once per step.

###  III-B Synthetic Power Analyses

In a traditional power analysis, real data is collected and used to determine
the sample size needed to achieve a desired experimental power. Let $p$ be the
p-value returned by a statistical test; denoting the probability of obtaining
a result equal to or more extreme than what is observed under the assumption
of no effect or no difference (null hypothesis). Let $\alpha$ be the pre-
determined test threshold; denoting the rate of falsely rejecting the null
hypothesis (type 1 error) threshold (often set to $.1$ in neuroimaging
studies). The power is denoted by $\gamma^{*}$, given by:

$\gamma^{*}=\mathbb{E}[\mathbbm{1}{\\{p<\alpha\\}}].$

In our simulated experiments, we mimic a power analysis with both real and
synthetic data by considering samples from the underlying distributions
$D_{1}$ and $D_{2}$. The experimental procedure is outlined in Algorithm 1.

Algorithm 1 Power Analyses for two-sample test

1: two data distributions $D_{1}$ and $D_{2}$; the number of samples
$n\in\\{N_{start},...,N_{end}\\}$; the number of trials $K$.

2:$\gamma^{*(N_{start},...,N_{end})}$

3:Train two generative models to recover estimates $\hat{D_{1}}$ and
$\hat{D_{2}}$ of the underlying data distributions $D_{1}$ and $D_{2}$.

4:for sample size $n=N_{start},...,N_{end}$ do

5: for trial $k=1,...,K$ do

6: Uniformly sample replicates of size $n$ from the real distributions $D_{1}$
and $D_{2}$ and the synthetic distributions $\hat{D_{1}}$ and $\hat{D_{2}}$.

7: Compute statistics for tests distinguishing between the real and synthetic
bootstrap replicates and calculate the $p$-values as $p^{real}_{n,k}$ and
$p^{syn}_{n,k}$.

8: Estimate the power $\gamma^{*(n)}_{real}$ of the test distinguishing
between $D_{1}$ and $D_{2}$ and $\gamma^{*(n)}_{syn}$ of the test
distinguishing between $\hat{D_{1}}$ and $\hat{D_{2}}$ with sample size $n$ by
$\gamma^{*(n)}=\frac{1}{K}\sum_{k}\mathbbm{1}{\\{p_{n,k}<\alpha\\}}$.

9: end for

10:end for

11:$\gamma^{*(N_{start},...,N_{end})}$

For simulated data, we consider various approaches to sampling and testing.
The first is direct sampling from the original distributions, which we denote
as resampling. Clearly, this is only possible with simulated data. We also
consider  boostrap resampling [31]. Here, we sample from the true
distributions once. Then, during each iteration, we need only sample a subset
from it. Finally, given the initial sample, we consider fitting the generative
model, followed by standard resampling from the synthetic distribution as a
substitute for the real data.

###  III-C Cognitive Neuroimaging Data

The fMRI ICW-GAN introduced in Sec. III-A can synthesize fMRI data conditioned
on cognitive process tag combinations that do not exist in the original
dataset. Thus we can formulate synthetic power analyses with data that has
never been collected before. To validate this approach, we provide details on
our experimental procedure in Algorithm 2.

Algorithm 2 Power Analyses for fMRI data

1: multi-tag fMRI dataset $D$; the number of samples
$n\in\\{N_{start},...,N_{end}\\}$; the number of trails $K$

2:$\gamma^{*(N_{start},...,N_{end})}$

3:Train a generative model on $D$ to recover an estimate $\hat{D}$ of the
underlying data distribution.

4:Split the dataset $D$ into two sub-samples $D=(D_{0},D_{1})$ where $D_{1}$
only contains samples with tag $x$ and $D_{0}$ contains the contrast tag (or
other images).

5:for sample size $n=N_{start},...,N_{end}$ do

6: for trial $k=1,...,K$ do

7: Draw samples from the generative model. $\hat{D}_{1}$ are synthetic sampled
conditioned on tag $x$, and $\hat{D}_{0}$ are synthetic examples without tag
$x$.

8: Conduct a statistical test between real data $D_{0}$ and $D_{1}$ to obtain
the $p$-value as $p^{real}_{n,k}$. Conduct a statistical test between
synthetic datasets $\hat{D}_{0}$ and $\hat{D}_{1}$to obtain the $p$-value as
$p^{syn}_{n,k}$.

9: Estimate the power $\gamma^{*(n)}_{real}$ of the test distinguishing
between $D_{0}$ and $D_{1}$ and the power $\gamma^{*(n)}_{syn}$ of the test
distinguishing between $\hat{D_{0}}$ and $\hat{D_{1}}$ with sample size $n$ by
$\gamma^{*(n)}=\frac{1}{K}\sum_{k}\mathbbm{1}{\\{p_{n,k}<\alpha\\}}$.

10: end for

11:end for

12:$\gamma^{*(N_{start},...,N_{end})}$

For the fMRI data whose true distribution is unknown, we apply the “bootstrap
sampling” method for real fMRI data. Similar to the settings in the simulated
data, the “resampling” strategy is used for synthetic data. As before, if at
every sample size the power of a test computed between $\hat{D_{0}}$ and
$\hat{D_{1}}$ is similar to the power of a test computed between $D_{0}$ and
$D_{1}$, then using synthetic data as a stand-in for real data will yield a
similar sample size estimation to what we would have achieved with real data.

##  IV Experiment

“Non-visual”

“Visual”

“Non-auditory”

“Auditory”

(a) Real images (b) Synthetic images

Figure 1: Visualization of real images (column (a)) and synthetic images
(column (b)). Conditional labels are also shown.

Next, we present our qualitative and quantitative evaluation of the proposed
synthetic power analyses on simulated data in Sec. IV-A and on fMRI data in
Sec. IV-B. We compare the power analysis results between synthetic and real
data using the classical two sample t-test and the non-parametric maximum mean
discrepancy (MMD) test with an L1 norm as the measure of distance [29].

For the fMRI brain images, we first present detailed qualitative results via
examples of generated 3D volumes using the ICW-GAN. Then, we demonstrate
prospective power analysis results for neuroimaging data.

###  IV-A Synthetic power on simulated data

We employ two multivariate Gaussian distributions,
$D_{1}\sim\mathcal{N}(\mu=\textbf{0},\Sigma=\mathbb{I})$ and
$D_{2}\sim\mathcal{N}(\mu=\textbf{0.3},\Sigma=\mathbb{I})$ with a dimension of
10. We vary the sample sizes from 0 to 500 with a step size of 20, i.e.,
$N_{start}=0$ and $N_{end}=500$. For every sample size, we repeat the
procedure 50 times to compute the power, i.e., $K=50$. We use two naive GAN
models to generate examples for the two distrubutions of the simulated data.
The structure of the generator and the critic is a standard multilayer
perceptron (MLP) network with 3 layers and ReLU activation between each layer.
The input of the GAN are sampled data points from $D_{1}$ and $D_{2}$. We
train the GAN models for 3,000 iterations with a batch size of 300. We
visualize the loss curves of the GANs and synthetic data in order to guarantee
that the GANs are well-trained. We compute power using a t-test and an MMD
test to measure the distance. We set the significance level $\alpha=0.05$.

Figure 2: Power curves on simulated data. The two multivariate Gaussian
distributions are $D_{1}\sim\mathcal{N}(\mu=\textbf{0},\Sigma=\mathbb{I})$ and
$D_{2}\sim\mathcal{N}(\mu=\textbf{0.3},\Sigma=\mathbb{I})$ with the dimension
of 10. The red line refers to $\gamma^{*}=0.8$.

Figure 2 summarizes the results of the t-test and MMD test power curves which
compare the two Gaussian distributions (the shaded region illustrates the
variances). We briefly outline some observations from the plots in the
following.

i) Using the “Resampling” strategy outperfoms the “Bootstrap” as expected.
Recall that at each iteration, “Resampling” obtains samples from the real
distributions, which one expects will result in a more powerful test.

ii) In most cases, the power curves of synthetic brain data are lower than the
curves of real brain data, suggesting that the test is appropriately
conservative in many cases.

###  IV-B Synthetic power on fMRI data

Neurovault[32] is currently the largest open database of preprocessed
neuroimaging data. We evaluate the performance of the proposed synthetic power
analyses on the the largest Neurovault functional brain image collection
1952111The collection is available here:
https://neurovault.org/collections/1952. Collection 1952 is a test-dependent
fMRI dataset with 19 meta-labels such as ‘visual,’ ‘language,’ and
‘calculate.’. The dimensions of the brain images in collection 1952 are
$53\times 63\times 46$. Following standard preprocessing procedures, a brain
image is normalized using the nilearn python
package222http://nilearn.github.io. We first project the data to 10 principle
components of fMRI volumes obtained from Principal Component Analysis (PCA)
for all the power analysis [33]. We conduct experiments for two labels having
the largest number of fMRI volumes in collection 1952, i.e., “visual” and
“auditory”. We train an ICW-GAN for each label. As an example, we divide
collection 1952 into two sets: with the tag “visual” and without “visual”. We
train an ICW-GAN for the two contrast labels of data. We conduct the same
procedure for the label “auditory”. Note that there could be an overlap across
different labels, e.g., an fMRI data can have both tags “visual” and
“auditory”.

Before the quantitative analysis, we visualize 2D projections of several real
and synthetic brain volumes with different tags in Fig. 1. We show real fMRI
data in Column (a) and synthetic images in Column (b). Each row represents
images with a certain tags, i.e., “visual” (first row) vs “non-visual”(second
row), and “auditory” (third row) vs “non-auditory”(forth row). Previous work
has examined the quality of synthetic images for data augmentation [7].
Specifically, classifiers are trained on the fMRI data with the two contrast
tags, and the results indicated that adding synthetic brain images to the
original real training dataset helped to improve the performance of
classification. All test parameters, e.g., $\alpha$ and the number of trials
are the same as for the simulated data.

Figure 3: Power curves for fMRI data with/without the tag “visual”. The red
line refer to $\gamma^{*}=0.8$, typically used by for neuroimaging
experiments. Figure 4: Power curves on fMRI data with/without the tag
“auditory”. The red line refers to $\gamma^{*}=0.8$, typically used for
neuroimaging experiments.

We present the power curves (curves are independently smoothed) for “visual”
and “auditory” tags in Fig. 3 and Fig. 4. From the two figures, we notice that
the “MMD bootstrap” curves are the highest performing, achieving 80% power
with fewer samples. The power curves of synthetic data are consistently lower
than those of real brain data on the same test, suggesting that the power
estimates are conservative (conservative power estimates ideal for sample size
selection). Our results recover the empirical observation from [26] that the
MMD test outperforms the t-test on data that does not satisfy Gaussian
distributional assumptions. For our synthetic brain images, the MMD
outperforms the t-test with sample sizes greater than 200. The curves of the
real images illustrate similar results.

##  V Conclusion

This work demonstrates the potential utility of synthetic power analyses
through experimental evaluation of simulated data and fMRI data. Specifically,
we compare the power analysis results of synthetic and real data and
illustrate that the GAN model sufficiently recovers the distributional
characteristics of the real data with respect to power estimates, which, in
turn, translates to accurate sample size estimates. We employ two statistical
testing methods, i.e., the t-test and the MMD test, and compare the power in
different cases. For imaging data, our results focus on the 3D Conditional
Improved Wasserstein Generative Adversarial Networks to synthesize diverse,
high-quality brain images. Our empirical results suggest that power analysis
results calculated with synthetic data are conservative compared to real data,
indicating that the synthetic powers could serve as a low-cost and
statistically reliable replacement for real data used in power analyses.

Synthetic power analyses have distinct advantages over traditional methods –
the simulated data are not restricted by sample size and can synthesize images
corresponding to flexible experimental conditions (by conditioning the
generative model). For future work, we plan on a more thorough evaluation of
synthetic power analysis, including evaluation in real experimental
conditions. We are also interested in theoretical analyses of the properties
of synthetic power analyses in different settings.

## References

  * [1] O. S. Collaboration _et al._ , “Estimating the reproducibility of psychological science,” _Science_ , 2015.
  * [2] H. Pashler and E.-J. Wagenmakers, “Editors’ introduction to the special section on replicability in psychological science: A crisis of confidence?” _Perspectives on Psychological Science_ , 2012.
  * [3] J. A. Mumford, “A power calculation guide for fmri studies,” _Social cognitive and affective neuroscience_ , 2012.
  * [4] J. Durnez, J. Degryse, B. Moerkerke _et al._ , “Power and sample size calculations for fmri studies based on the prevalence of active peaks.” _bioRxiv_ , 2016.
  * [5] L. Glover, “Why does an mri cost so darn much? — money,” 2014. [Online]. Available: http://time.com/money/2995166/why-does-mri-cost-so-much/
  * [6] “University of michigan functional mri laboratory.” [Online]. Available: http://fmri.research.umich.edu/users/billing.php
  * [7] P. Zhuang, A. G. Schwing, and O. Koyejo, “Fmri data augmentation via synthesis,” 2019.
  * [8] A. Gretton, K. M. Borgwardt, M. J. Rasch _et al._ , “A kernel two-sample test,” _Journal of Machine Learning Research_ , 2012.
  * [9] R. A. Poldrack and K. J. Gorgolewski, “Making big data open: data sharing in neuroimaging,” _Nature neuroscience_ , 2014.
  * [10] G. Varoquaux and B. Thirion, “How machine learning is shaping cognitive neuroimaging,” _GigaScience_ , 2014.
  * [11] D. D. Cox and R.-L. Savoy, “Functional magnetic resonance imaging (fMRI)“brain reading”: detecting and classifying distributed patterns of fMRI activity in human visual cortex,” _Neuroimage_ , 2003.
  * [12] F. Pereira, T. Mitchell, and M. Botvinick, “Machine learning classifiers and fMRI: a tutorial overview,” _Neuroimage_ , 2009.
  * [13] D. Nathawani, T. Sharma, and Y. Yang, “Neuroscience meets deep learning,” 2016\.
  * [14] O. Firat, L. Oztekin, and F. T. Y. Vural, “Deep learning for brain decoding,” in _Image Processing (ICIP), 2014 IEEE International Conference on_. IEEE, 2014.
  * [15] S. Koyamada, Y. Shikauchi, K. Nakae _et al._ , “Deep learning of fmri big data: a novel approach to subject-transfer decoding,” _arXiv preprint arXiv:1502.00093_ , 2015.
  * [16] M. Svanera, S. Benini, G. Raz _et al._ , “Deep driven fmri decoding of visual categories,” _arXiv preprint arXiv:1701.02133_ , 2017.
  * [17] I. Goodfellow, J. Pouget-Abadie, M. Mirza _et al._ , “Generative adversarial nets,” in _Advances in neural information processing systems_ , 2014.
  * [18] M. Mirza and S. Osindero, “Conditional generative adversarial nets,” _arXiv preprint arXiv:1411.1784_ , 2014.
  * [19] M. Arjovsky, S. Chintala, and L. Bottou, “Wasserstein gan,” _arXiv preprint arXiv:1701.07875_ , 2017.
  * [20] I. Gulrajani, F. Ahmed, M. Arjovsky _et al._ , “Improved training of wasserstein gans,” in _Advances in Neural Information Processing Systems_ , 2017.
  * [21] G. D. Ruxton, “The unequal variance t-test is an underused alternative to student’s t-test and the mann–whitney u test,” _Behavioral Ecology_ , vol. 17, no. 4, pp. 688–690, 2006.
  * [22] N. Cressie and H. Whitford, “How to use the two sample t-test,” _Biometrical Journal_ , vol. 28, no. 2, pp. 131–148, 1986.
  * [23] H. Keselman, A. R. Othman, R. R. Wilcox, and K. Fradette, “The new and improved two-sample t test,” _Psychological Science_ , vol. 15, no. 1, pp. 47–51, 2004.
  * [24] H. O. Posten, “Robustness of the two-sample t-test,” in _Robustness of statistical methods and nonparametric statistics_. Springer, 1984.
  * [25] A. Gretton, K. Borgwardt, M. Rasch _et al._ , “A kernel method for the two-sample-problem,” _In Advances in neural information processing systems_ , 2007.
  * [26] A. Gretton, K. Borgwardt, M. Rasch, B. Scholkopf _et al._ , “A kernel two-sample test.” _Journal of Machine Learning Research, 13(Mar), pp.723-773._ , 2013.
  * [27] K. P. Chwialkowski, A. Ramdas, D. Sejdinovic _et al._ , “Fast two-sample testing with analytic representations of probability measures,” in _Advances in Neural Information Processing Systems_ , 2015, pp. 1981–1989.
  * [28] A. Smola, A. Gretton, L. Song _et al._ , “A hilbert space embedding for distributions,” in _International Conference on Algorithmic Learning Theory_ , 2007.
  * [29] M. Scetbon and G. Varoquaux, “Comparing distributions: $l1$ geometry improves kernel two-sample testing,” _arXiv preprint arXiv:1909.09264_ , 2019\.
  * [30] T. Tieleman and G. Hinton, “Divide the gradient by a running average of its recent magnitude,” 2012.
  * [31] B. Efron and R. J. Tibshirani, _An introduction to the bootstrap_. CRC press, 1994.
  * [32] K. J. Gorgolewski, G. Varoquaux, G. Rivera _et al._ , “Neurovault. org: a web-based repository for collecting and sharing unthresholded statistical maps of the human brain,” _Frontiers in neuroinformatics_ , 2015.
  * [33] K. Pearson, “Liii. on lines and planes of closest fit to systems of points in space,” _The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science_ , 1901.