# Population Power Curves in ASCA with Permutation Testing

José Camacho111Research Centre for Information and Communication Technologies
(CITIC-UGR), University of Granada, Spain Michael Sorochan
Armstrong222Research Centre for Information and Communication Technologies
(CITIC-UGR), University of Granada, Spain<EMAIL_ADDRESS>
###### Abstract

In this paper, we revisit the Power Curves in ANOVA Simultaneous Component
Analysis (ASCA) based on permutation testing, and introduce the Population
Curves derived from population parameters describing the relative effect among
factors and interactions. We distinguish Relative from Absolute Population
Curves, where the former represent statistical power in terms of the
normalized effect size between structure and noise, and the latter in terms of
the sample size. Relative Population Curves are useful to find the optimal
ASCA model (e.g., fixed/random factors, crossed/nested relationships,
interactions, the test statistic, transformations, etc.) for the analysis of
an experimental design at hand. Absolute Population Curves are useful to
determine the sample size and the optimal number of levels for each factor
during the planning phase on an experiment. We illustrate both types of curves
through simulation. We expect Population Curves to become the go-to approach
to plan the optimal analysis pipeline and the required sample size in an omics
study analyzed with ASCA.

###### keywords:

ANOVA Simultaneous Component Analysis, Power Curves, Sample Size, Effect Size,
Multivariate ANOVA.

##  1 Introduction

Prof. Smilde’s research group proposed the ANOVA-Simultaneous Component
Analysis (ASCA) [1], a powerful framework for analysing the individual
influence of different experimental factors and their interactions in
experimental designs with a high number of responses. ASCA represents a
natural multivariate extension of the Analysis of Variance (ANOVA). The theory
associated to ANOVA is extensive and complex [2], and several of its
developments have been incrementally incorporated into the ASCA framework
through the years [3, 4, 5]. Still, ASCA can be considered a developing
technique, and many unresolved questions remain regarding best practices [6,
7].

One such questions is how to derive a power analysis in the context of ASCA.
Statistical power is a relevant concept in inferential statistics. The power
of a test measures the probability that it correctly rejects the null
hypothesis ($H_{0}$) when the alternative hypothesis ($H_{1}$) is true. The
power can be defined as $1-\beta$ for $\beta$ the Type II Error (false
negative) probability. Power Curves are a form of power analysis where power
is represented in terms of the sample size, that is, of the number of
replicates or experimental runs under the same conditions. This is a
recommended analysis prior to any multivariate experiment (e.g., a clinical
study with omics responses) to determine the required number of subjects and
experimental levels for a desired statistical power.

In standard univariate ANOVA, or when relatively few responses are considered,
it is possible to use analytical methods to derive Power Curves based on
typical assumptions typical (such as normality) [8]. However, in the context
of more than just a few responses, or to assess any potential violation of
assumptions, numerical methods [9] are a viable alternative. Arguably the most
popular approach for statistical inference in ASCA is permutation testing [9,
10]: a resampling method that transforms ASCA into a distribution-free
approach, that is, more flexible than parametric ANOVA. The derivation of
Power Curves based on ASCA’s permutation testing necessarily requires
numerical approaches.

In previous work [6], we introduced the simulated Power Curves in ASCA as a
strategy to optimize an ASCA pipeline for a specific experimental design in
terms of statistical power. Simulated Power Curves can be used to find the
optimal ASCA model in terms of fixed/random factors, crossed/nested
relationships, interactions, test statistic, transformations, and others. Our
approach was defined to compare several models in terms of relative power, but
it cannot be used to make sample size estimations for an entirely new
multivariate experiment. In this paper, we revisit the simulated Power Curve
approach to generalize it; In particular, we make the following contributions:

  * 1.

We generalize ASCA Power Curves so that any design can be simulated, including
very complex relationships among factors and interactions.

  * 2.

We define the Population Curves, derived from population parameters (standard
deviations) describing the relative effect among factors and interactions.

  * 3.

We propose two types of Population Curves:

    * (a)

Relative Population Curves represent statistical power in terms of the
relative effect size between structure and noise. They are useful to optimize
the ASCA pipeline for an analysis at hand.

    * (b)

Absolute Population Curves represent statistical power in terms of the sample
size. They are useful to plan ahead the number of replicates and/or levels to
use in a designed study.

  * 4.

We illustrate the behaviour of the two types of Population Curves through
simulation.

  * 5.

We provide open software for the generation of Population Curves and for the
replication of the results in this paper. Relative and Absolute Population
Curves can be computed with ‘powercurve’ routine in the MEDA Toolbox stable
release v1.4333Stable release of MEDA Toolbox v1.4.
https://github.com/CoDaSLab/MEDA-Toolbox/releases/tag/v1.4.

The rest of the paper is organized as follows: Section 2 introduces the ASCA
framework. Section 3 discusses the concept of Power Curve from a theoretical
perspective. Sections 4 and 5 present the algorithms to compute Relative and
Absolute Population Curves, respectively. Section 6 discusses simulation
results and Section 7 draws the conclusions of the work.

##  2 ANOVA Simultaneous Component Analysis

ASCA, like ANOVA, is mostly concerned with the analysis of data coming from an
experimental design. Following ANOVA, a common ASCA pipeline follows three
steps: (1) factorization of the data according to the experimental design; (2)
significance testing for factors and interactions; (3) visualization of
significant factors’ and interactions’ effects using Principal Component
Analysis (PCA) to understand separability among levels.

###  2.1 Factorization of the data

Let $\mathbf{X}$ be an $N\times M$ data matrix with $N$ the number of
experimental runs and $M$ the number of responses in a designed experiment.
Without loss of generality, we consider here the case of a design with two
crossed factors $A$ and $B$, their interaction, and an additional factor
$C(A)$ nested in $A$. This is a common configuration in multiple omics
experiments [11, 4, 5], useful to correct for the (often large) individual
variability ($C(A)$) and so increase the statistical power of the test. The
data in $\mathbf{X}$ can be decomposed as:

$\mathbf{X}=\mathbf{1}\mathbf{m}^{\mathrm{T}}+\mathbf{A}+\mathbf{B}+\mathbf{C}(\mathbf{A})+\mathbf{A}\mathbf{B}+\mathbf{E}$
(1)

where $\mathbf{1}$ is a vector of ones of suitable length ($N\times 1$),
$\mathbf{m}$ ($M\times 1$) denoting a vector containing the intercepts of the
$M$ measured variables, $\mathbf{A}$, $\mathbf{B}$ and
$\mathbf{C}(\mathbf{A})$ represent the factor matrices, $\mathbf{A}\mathbf{B}$
is the interaction matrix, and $\mathbf{E}$ is the residual matrix, all of
similar dimensions of $\mathbf{X}$.

To compute the factorization, we use the technique referred to as ASCA+ [3] to
account for mild unbalancedness in the data. We intentionally avoid ASCA
alternatives based on Linear Mixed Models (LMM) [4, 5] due to the large
increase of computational demand, prohibitive in the context of Power Curves.
In ASCA+, the decomposition is derived as the least squares solution of a
regression problem, where $\mathbf{X}$ is regressed onto a coding matrix
$\mathbf{D}$ as:

$\mathbf{X}=\mathbf{D}\boldsymbol{\Theta}+\mathbf{E}=\mathbf{1}\boldsymbol{\theta}+\mathbf{D}_{A}\boldsymbol{\Theta}_{A}+\mathbf{D}_{B}\boldsymbol{\Theta}_{B}+\mathbf{D}_{C(A)}\boldsymbol{\Theta}_{C(A)}+\mathbf{D}_{AB}\boldsymbol{\Theta}_{AB}+\mathbf{E}$
(2)

and $\mathbf{D}$ is constructed using sum coding [3] or another alternative
coding schemes [5] and $\boldsymbol{\Theta}$ and $\mathbf{E}$ are obtained
from:

$\boldsymbol{\Theta}=(\mathbf{D}^{\mathrm{T}}\mathbf{D})^{-1}\mathbf{D}\mathbf{X}$
(3) $\mathbf{E}=\mathbf{X}-\mathbf{D}\boldsymbol{\Theta}$ (4)

The encoding of experimental factors in the design matrix is an especially
important consideration, as it affects what information is passed through to
variance explained by the model, versus residual variance–this in turn affects
the apparent evidence for statistical significance.

###  2.2 Statistical significance testing

In this step, we test the statistical significance of each factor and
interaction. A widely used approach for ASCA inference is permutation testing
[9, 10]. Permutation testing can be performed by randomly shuffling the rows
of $\mathbf{X}$ in Equation (3), yielding a new set of regression
coefficients, and so a new factorization:

$\boldsymbol{\Theta}^{*}=(\mathbf{D}^{\mathrm{T}}\mathbf{D})^{-1}\mathbf{D}^{\mathrm{T}}\mathbf{X}^{*}$
(5) $\mathbf{E}^{*}=\mathbf{X}^{*}-\mathbf{D}\boldsymbol{\Theta}^{*}$ (6)

Permutation tests are carried out by comparing a given statistic, computed
after the ASCA factorization, with the corresponding statistic computed from
hundreds or more permutations of the rows in the observational data,
$\mathbf{X}$. The $p$-value is obtained as444Throughout the paper, we assume
that the higher the statistic the more significant the effect of the
factor/interaction.

$p=\frac{\\#\\{S_{k}^{*}\geq S;k=1,\ldots,K\\}+1}{K+1}$ (7)

where $S$ refers to the statistic computed from the true factorization,
$S_{k}^{*}$ is the statistic corresponding to the $k$-th random permutation,
$\\#\\{cond\\}$ refers to the number of times condition $cond$ is met, and $K$
is the total number of permutations. See [6] for a recent review on the
permutation approach and the relevance of the chosen statistic.

###  2.3 Post-hoc visualization

Significance testing in ANOVA/ASCA reveals the statistical significance of
factors’ and interactions’ effects, but not the specific levels (or
combination thereof). To identify significant differences across levels, post-
hoc tests are typically employed. The equivalent to a post-hoc visuzalition in
ASCA is the PCA visualization following Zwanenburg _et al._ [12], where the
PCA loadings are computed from the factor/interaction matrix, and score plots
are built from the sum of this matrix and the residuals. Following this
approach, score plots provide a visual comparison between the main/interaction
effects and the natural variability in the residuals. Othe approaches to PCA
can also be used [13].

ASCA is a supervised method, and as such it can suffer of “overfitting” like
any other regression and/or classification approaches [14]. ASCA relies in
permutation testing to avoid over-fitting. Thus, only statistically
significant factors/interactions should be visualized post-hoc with PCA.

##  3 Power Curves

Figure 1 illustrates the concept of a power curve as a comparison between two
distributions: the null distribution $P(t|H_{0})$ and the alternative
distribution $P(t|H_{1})$. In the figure, both distributions are assumed to be
normal, but in a real situation this does not have to be the case. The shaded
areas represent the probability of error, associated with an observation
incorrectly rejecting the null hypothesis through random chance (type I
error), with probability $\alpha$ (in blue), versus the probability of an
observation incorrectly not rejecting the null hypothesis (type II error),
with probability $\beta$ (in red). Larger effect sizes $\theta$, manifesting
as wider separations between both distributions, increase the statistical
power $1-\beta$ (reducing the type II error) for a fixed value of $\alpha$.
Alternatively, for both fixed $\alpha$ and the effect size $\theta$,
increasing the sampling size reduces the variance of the distributions
$P(t|H_{0})$ and $P(t|H_{1})$, which leads to a reduction of overlapping areas
as $1-\beta$ increases.

Figure 1: Illustration of effect size ($f(\theta)$) versus the probability of
a type-I error, $\alpha$, and the probability of a type II error, $\beta$.
Statistical power is defined as 1 - the false negative (type-II) error rate.

##  4 Relative Population Curves for ASCA

Relative Population Curves or RPCs are derived from population parameters
describing the standard deviation in factors and interactions and represent
statistical power in terms of the relative effect size between structure and
noise.

To generate RPCs we generalize the approach by Camacho _et al._ [6]. We
simulate data that progressively increases the relative effect size $\theta$
and therefore the power (Figure 1). In the following, we use model (1) to
showcase the RPCs, since this model includes crossed relationships,
interactions (between $A$ and $B$) and nested relationships (between $A$ and
$C(A)$), so that almost any other model can be derived from it. We can also
think of this model as an illustration of factors/interactions organized in
different orders (Figure 2), as discussed by Anderson and Ter Braak [9] and in
the Hasse diagrams by Marini _et al._ [15].

Figure 2: Ordering structure among factors and interaction in model (1).

Our approach to generate RPCs follows these steps:

  * 0.

INPUT:

    * (a)

$\mathbf{F}$ the design matrix

    * (b)

$H$ a model hierarchical structure (as illustrated by a Hassel diagram) with
information about factors, crossed/nested relationships and their
interactions. In this model structure, a factor $f$ can have descendants
(nested factors and associated interactions) and ancestors (factors in which
$f$ is nested). An interaction $i$ can also have descendants (other higher-
order interactions) and ancestors (the factors included in the interaction).
In model (1) and Figure 2, $A$ is ancestor of $C(A)$ and $A$ and $B$ are
ancestors of $AB$.

    * (c)

$M$ the number of responses

    * (d)

$k_{f}$ for $f\in\\{1...F\\}$, $k_{i}$ for $i\in\\{1...I\\}$, and $k_{e}$ the
coefficients with the standard deviation555Given the multivariate nature of
the response, and to simplify notation, we generally refer to the standard
deviation $\sigma$ of each response vector, so that the expected standard
deviation of each individual response would be $\sigma/\sqrt{M}$ for factors
and the interactions, and the residuals

    * (e)

$R$ the number of repetitions to generate a statistically representative RPC

    * (f)

$P$ the number of permutations (in each repetition)

    * (g)

$\delta$ incremental steps in the effect size

    * (h)

$\alpha$ the imposed probability of falsely rejecting the null hypothesis
(i.e. the significance level)

  * 1.

Set $power_{f}(\theta)=0$ and $power_{i}(\theta)=0$ for each factor $f$ and
interaction $i$, respectively, and for $\theta$ from 0 to $10\delta$ in
$\delta$ steps; and $N$ is set to the number of rows of $\mathbf{F}$.

  * 2.

For each repetition from 1 to $R$

    * 2.1.

Generate random matrices to represent level/cell averages

      * 2.1.1.

For each factor $f_{1}$ of order 1 in $H$, and so with no ancestors (e.g., $A$
and $B$ in model (1)), count the number of levels $L_{f_{1}}$ in matrix
$\mathbf{F}$ and simulate:

$\mathbf{\bar{X}}_{f_{1}}(L_{f_{1}},M)\sim\mathcal{D}_{f_{1}}$ (8)

where $\mathcal{D}_{f_{1}}$ is a pseudo-random number generator (PRNG),
potentially based on the normal distribution or other distribution that may
deviate moderately (e.g., uniform) or severely (e.g., exp3) from normality
[9].

      * 2.1.2.

For each factor $f$ in $H$ with ancestor factor(s) $f_{a}$, for
$a={1...A_{f}}$ (e.g., $C(A)$ nested on $A$ in model (1)):

$L_{f}=r_{f}\cdot\prod_{a=1}^{A_{f}}L_{f_{a}}$ (9)
$\mathbf{\bar{X}}_{f}(L_{f},M)\sim\mathcal{D}_{f}$ (10)

with $r_{f}$ the number of replicates in each unique combination of levels
(cell), and $\mathcal{D}_{f}$ the chosen PRNG.

      * 2.1.3.

For each interaction $i$ in $H$ with ancestor factor(s) $f_{a}$ for
$a={1...A_{i}}$ (e.g., $AB$ in model (1)):

$L_{i}=\prod_{a=1}^{A_{i}}L_{i_{a}}$ (11)
$\mathbf{\bar{X}}_{i}(L_{i},M)\sim\mathcal{D}_{i}$ (12)

with $\mathcal{D}_{i}$ the chosen PRNG.

    * 2.2.

Generate background variability for a chosen PRNG:

$\mathbf{X}_{E}(N,M)\sim\mathcal{D}_{E}$ (13)

    * 2.3.

Normalize each matrix $\mathbf{\bar{X}}_{f}$ for each factor $f$,
$\mathbf{\bar{X}}_{i}$ for each interaction $i$, and $\mathbf{X}_{E}$ so that
the Frobenius norm equals the squared root of the number of rows666This
normalization is instrumental for the correspondence of theoretical and
numerical results, as shown later on..

    * 2.4.

For each observation $n$ from 1 to $N$:

      * 2.4.1.

For each factor $f$, we build $\mathbf{X}_{f}(N,M)$ from
$\mathbf{\bar{X}}_{f}(L_{f},M)$ and the design matrix $\mathbf{F}$:

$\mathbf{x}_{f}^{n}=\mathbf{\bar{x}}^{l}_{f}$ (14)

with $\mathbf{x}_{f}^{n}$ the $n$-th row of $\mathbf{X}_{f}(N,M)$ and
$\mathbf{\bar{x}}^{l}_{f}$ the $l$-th row of $\mathbf{\bar{X}}_{f}(L_{f},M)$,
with $l$ determined according to matrix $\mathbf{F}$.

      * 2.4.2.

For each interaction $i$, we build $\mathbf{X}_{i}(N,M)$ from
$\mathbf{\bar{X}}_{i}(L_{i},M)$ and the design matrix $\mathbf{F}$:

$\mathbf{x}_{i}^{n}=\mathbf{\bar{x}}^{l}_{i}$ (15)

with $\mathbf{x}_{i}^{n}$ the $n$-th row of $\mathbf{X}_{i}(N,M)$ and
$\mathbf{\bar{x}}^{l}_{i}$ the $l$-th row of $\mathbf{\bar{X}}_{i}(L_{i},M)$,
with $l$ determined according to matrix $\mathbf{F}$.

    * 2.5.

Compute the matrices with the structural and residual part with the standard
deviation coefficients:

$\mathbf{X}_{S}=\sum_{f}k_{f}\mathbf{X}_{f}+\sum_{i}k_{i}\mathbf{X}_{i}$ (16)
$\mathbf{X}_{E}=k_{e}\mathbf{X}_{E}$ (17)

    * 2.6.

For $\theta$ from 0 to $10\delta$ in $\delta$ steps:

      * 2.6.1.

Yield the simulated data:

$\mathbf{X}=\theta\mathbf{X}_{S}+(1-\theta)\mathbf{X}_{E}$ (18)

      * 2.6.2.

Compute ASCA+ partition and the F-ratio for factors and interactions for both
the simulated data and $P$ permutations. For (high-order) factors and
interactions in $H$ with no descendants:

$F_{f}=(SS_{f}/DoF_{f})/(SS_{E}/DoF_{E})$ (19)
$F_{i}=(SS_{i}/DoF_{i})/(SS_{E}/DoF_{E})$ (20)

where $SS$ refers to the sum-of-squares (the Frobenius norm) of a
factor/interaction/residual matrix in the factorization with ASCA+, and $DoFs$
represents the corresponding degrees of freedom [2]. The DoFs of a factor is
the number of levels minus one, the DoFs of an interaction is the product of
the DoFs of its factors, and the DoFs of the residuals is the total (number of
observations minus 1 in the data) minus the DoFs of all factors and
interactions in the model. The ratio between the $SS$ and the $DoF$ is often
called the mean sum-of-squares ($MS$). For any factors and interactions in $H$
with descendants:

$F_{f}=\frac{(SS_{f}/DoF_{f})}{(\sum_{d=1}^{D_{f}}SS_{f_{d}}+\sum_{d=1}^{D_{i}}SS_{i_{d}})/(\sum_{d=1}^{D_{f}}DoF_{f_{d}}+\sum_{d=1}^{D_{i}}DoF_{i_{d}})}$
(21)
$F_{i}=\frac{(SS_{i}/DoF_{i})}{(\sum_{d=1}^{D_{i}}SS_{i_{d}})/(\sum_{d=1}^{D_{i}}DoF_{i_{d}}))}$
(22)

with descendant factor(s) $f_{d}$ for $d={1...D_{f}}$ and descendant
interaction(s) $f_{i}$ for $d={1...D_{i}}$.

      * 2.6.3.

For each factor $f$, if the associated p-value computed using Eq. (7) is below
$\alpha$ do:

$power_{f}(\theta)=power_{f}(\theta)+1$ (23)

      * 2.6.4.

For each factor $i$, if the associated p-value computed using Eq. (7) is below
$\alpha$ do:

$power_{i}(\theta)=power_{i}(\theta)+1$ (24)

  * 3.

Normalize $power_{f}(\theta)=power_{f}(\theta)/R$ and
$power_{i}(\theta)=power_{i}(\theta)/R$ for each factor $f$ / interaction $i$

The algorithm works as follows. In step 0, we set the general characteristics
of the data simulation. In step 1, the algorithm initializes the values in the
RPC. In step 2, we iterate through a number of repetitions to compute the RPC.
Each repetition consists on the simulation of a structural part (in
$\mathbf{X}_{S}$) and a residual part (in $\mathbf{X}_{E}$). The inner loop
builds the data for intermediate cases between the absence of effect
($\mathbf{X}=\mathbf{X}_{E}$) and absence of residuals
($\mathbf{X}=\mathbf{X}_{S}$), factorizes it with ASCA+ and performs the
statistical inference through permutation testing. If the computed
significance is below the imposed significance level, the power is increased
in one. In step 3, the power is normalized by the number of repetitions.

Some alternative configurations for a RPC that are relevant in practice can be
straightforwardly implemented by modifying specific parameters or small parts
of the algorithm:

  * 1.

Non-balanced designs can be easily integrated in the design matrix
$\mathbf{F}$, see [6].

  * 2.

By properly choosing $\mathcal{D}_{E}$, we can emulate different distributions
in the residuals to assess robustness in front of deviations from normality in
the manner of Anderson and Ter Braak [9] but for multivariate responses. We
can also generate the level averages using different distributions.

  * 3.

Both fixed and random factors can be simulated in the same manner.

  * 4.

We can integrate complex designs by adding multiple crossed and nested
relationships as well as interactions.

  * 5.

We can generate RPCs for a specific factor or interaction in an experimental
design, or for all of them. The algorithm provides the solution for a RPC that
considers a simultaneous incremental effect in all factors and interactions of
the model, but some factors may be deactivated by setting the corresponding
coefficient to 0. We can also maintain the effect of a significant
factor/interaction fixed along the curve by adding its contribution directly
in Eq. (18), for example:

$\mathbf{X}=\theta\mathbf{X}_{S}+(1-\theta)\mathbf{X}_{E}+k_{f}\mathbf{X}_{f}$
(25)

  * 6.

In a similar way, we can add a covariate
$\mathbf{X}_{cv}(N,M)\sim\mathcal{D}_{cv}$ directly in Eq. (18), for example:

$\mathbf{X}=\theta\mathbf{X}_{S}+(1-\theta)\mathbf{X}_{E}+k_{cv}\mathbf{X}_{cv}$
(26)

  * 7.

We can generate RPCs for alternative statistics, see [6].

##  5 Absolute Population Curves for ASCA

Absolute Population Curves (APCs) differ to RPCs in that the power is shown in
terms of the sampling size, rather than the relative effect size. APCs have
the same applications as the RPCs, but with the additional advantage that they
provide information about the number of replicates and/or factor levels that
one may use in a multivariate experiment in order to attain a given
probability of success, that is, the probability of rejecting the null
hypothesis when the alternative hypothesis is true: the power $1-\beta$.

An APC is built by simulating data that progressively enlarges the number of
levels in a specific factor, or the whole experimental design. The consequence
of this enlargement is a reduction of the variance of both the null and the
alternative distributions (Figure 1), with a subsequent increase of
statistical power. For instance, if we take model (1), we can apply APCs to
investigate how the statistical power of the factors and the interaction is
affected when:

  * 1.

We iteratively enlarge the number of levels in $A$. Often $A$ is a factor that
controls the number of groups of individuals in a clinical study, e.g., with a
number of disease sub-types. This APC gives useful information about whether
incorporating more or less sub-types can impact our probability of success.

  * 2.

We iteratively enlarge the number of levels in $B$. Often $B$ is a factor with
several repeated measures over the same individual, e.g., in time or in
biological samples. Then, the APC will allow us to investigate the effect of
adding more time points/biological samples in our probability of success.

  * 3.

We iteratively enlarge the number of levels in $C(A)$. Often $C(A)$ models the
individual variability. The APC will give us an idea about the number of
individuals per group we should choose for a certain probability of success.

  * 4.

We iterative enlarge the whole experiment. The APC will give us an idea about
the general number of replicates we should choose for a certain probability of
success. This is often a good alternative choice when a factor like $C(A)$ is
not in the design.

Our approach to generate APCs follows these steps (the algorithm is summarized
not to replicate detailed explanations of the RPC algorithm)

  * 0.

INPUT:

    * (a)

INPUTs in the RPC algorithm

    * (b)

$\theta$ the fixed effect size

    * (c)

$f_{rep}$ the index of the factor that is replicated, or 0 if all the whole
experiment is replicated

    * (d)

$\delta$ incremental steps in the sampling size $\eta$

  * 1.

Set $power_{f}(\eta)=0$ and $power_{i}(\eta)=0$ for each factor $f$ and
interaction $i$, respectively, and for $\eta$ from 1 to $10\delta$ in $\delta$
steps.

  * 2.

For each repetition from 1 to $R$

    * 2.1.

For $\sigma$ from 1 to $10\delta$ in $\delta$ steps:

      * 2.1.1

Create $\mathbf{F}_{\theta}$ from $\mathbf{F}$ and $f_{rep}$; and $N$ is set
to the number of rows of $\mathbf{F}$.

      * 2.1.2

Generate random matrices to represent level averages in factors and
interactions.

      * 2.2.3

Generate background variability for a chosen PRNG.

      * 2.2.4

Normalize each matrix $\mathbf{\bar{X}}_{f}$ for each factor $f$,
$\mathbf{\bar{X}}_{i}$ for each interaction $i$, and $\mathbf{X}_{E}$ so that
the Frobenius norm equals the squared root of the number of rows.

      * 2.2.5

Compute final factor and interaction matrices of $N$ rows.

      * 2.2.6

Compute the matrices with the structural and residual part with the standard
deviation coefficients.

      * 2.2.7

Yield the simulated data.

      * 2.2.8

Compute ASCA+ partition and the F-ratio for factors and interactions for both
the simulated data and $P$ permutations.

      * 2.2.9

Update the power of factors and interactions.

  * 3.

Normalize $power_{f}(\eta)=power_{f}(\eta)/R$ and
$power_{i}(\eta)=power_{i}(\eta)/R$ for each factor $f$ / interaction $i$

The algorithm works as follows. In step 0 we set the general characteristics
of the data simulation, and in step 1 the algorithm initializes the values in
the APC. In step 2, we iterate through a number of repetitions to compute the
APC. Each repetition consists on an inner loop that progressively increases
the sampling size by either increasing in one the number of levels of a given
factor (for $f_{rep}>0$) or by adding one complete set of experimental runs.
Subsequently, the same simulation approach as in an RPC is followed to
generate the simulated matrix with both structural and residuals parts. This
matrix is factorized with ASCA+ and the statistical inference is performed
through permutation testing. If the computed significance is below the imposed
significance level, the power is increased in one. In step 3, the power is
normalized by the number of repetitions.

##  6 Simulation examples

###  6.1 Relative Population Curves

Figure 3 presents an example of the RPCs for model (1) and for a full
factorial design with four levels in $A$, three levels in $B$ and four
individuals in each cell of $C(A)$. The RPCs are shown with $95\%$ confidence
intervals computed by bootstrapping. In the example, all standard deviation
coefficients are fixed to 1/10 of the standard deviation in the residuals
($k_{A}=k_{B}=k_{C(A)}=k_{AB}=0.1$ and $k_{E}=1$). The behaviour of the RPCs
is the one expected for a correct power curve [6]: a) in the absence of effect
(i.e., at $\theta=0$), all curves adjust to the significance level of
$\alpha=0.05$; and b) at some given effect size, the curves start gaining
power until they reach 1.

There are some additional interesting patterns reflected in the relative power
of the curves, that is, in the order in which the curves start increasing and
finally reach 1. To understand this order in the curves, we can resort to
ANOVA theory. Given that all factors are derived from a (pseudo-)random
distribution, and following Montgomery [2], we can derive the relationships
between population variances and expected MSs in model (1):

$E(MS_{E})=\sigma^{2}_{E}$ (27)
$E(MS_{AB})=\sigma^{2}_{E}+r_{C(A)}\cdot\sigma_{AB}^{2}$ (28)
$E(MS_{C(A)})=\sigma^{2}_{E}+L_{B}\cdot\sigma^{2}_{C(A)}$ (29)
$E(MS_{B})=\sigma^{2}_{E}+r_{C(A)}\cdot\sigma_{AB}^{2}+L_{A}\cdot
r_{C(A)}\cdot\sigma_{B}^{2}$ (30)
$E(MS_{A})=\sigma^{2}_{E}+r_{C(A)}\cdot\sigma_{AB}^{2}+L_{B}\cdot\sigma^{2}_{C(A)}+L_{B}\cdot
r_{C(A)}\cdot\sigma_{A}^{2}$ (31)

where $\sigma^{2}_{A}$, $\sigma^{2}_{B}$, $\sigma^{2}_{C(A)}$,
$\sigma^{2}_{AB}$, and $\sigma^{2}_{E}$ are the population variance of the
factors, the interaction and the residuals, respectively, and in our example
we have $r_{C(A)}=4$ replicates, and number of levels $L_{A}=4$ and $L_{B}=3$.

Figure 3: Example of Relative Population Curve for model (1). The design
matrix $\mathbf{F}$ contains a full factorial design with four levels in $A$,
three levels in $B$ and four individuals in each cell of $C(A)$. Other inputs
are $M=400$, $k_{A}=k_{B}=k_{C(A)}=k_{AB}=0.1$, and $k_{E}=1$, $R=1000$,
$P=200$, $\delta=0.1$ and $\alpha=0.05$. We marked $\theta=0.5$ as a reference
for the following figures.

Combining previous equations and Eqs. (19) to (21), the inference statistics
follow:

$E(F_{A})=\frac{E(MS_{A})}{(DoF_{C(A)}E(MS_{C(A)})+DoF_{AB}E(MS_{AB}))/(DoF_{C(A)}+DoF_{AB})}=\frac{(\sigma^{2}_{E}+r_{C(A)}\cdot\sigma_{AB}^{2}+L_{B}\cdot\sigma^{2}_{C(A)}+L_{B}\cdot
r_{C(A)}\cdot\sigma_{A}^{2})(r_{C(A)}+L_{B}-1)}{r_{C(A)}(\sigma^{2}_{E}+L_{B}\cdot\sigma^{2}_{C(A)})+(L_{B}-1)(\sigma^{2}_{E}+r_{C(A)}\cdot\sigma_{AB}^{2})}$
(32) $E(F_{B})=\frac{E(MS_{B})}{E(MS_{AB})}=1+\frac{L_{A}\cdot
r_{C(A)}\cdot\sigma_{B}^{2}}{\sigma^{2}_{E}+r_{C(A)}\cdot\sigma_{AB}^{2}}$
(33)
$E(F_{C}(A))=\frac{E(MS_{C(A)})}{E(MS_{E})}=1+\frac{L_{B}\cdot\sigma^{2}_{C(A)}}{\sigma^{2}_{E}}$
(34)
$E(F_{AB})=\frac{E(MS_{AB})}{E(MS_{E})}=1+\frac{r_{C(A)}\cdot\sigma_{AB}^{2}}{\sigma^{2}_{E}}$
(35)

Adjusting the population variances to the square of the standard deviation
coefficients used in the RPCs of Figure 3, i.e.,
$k^{2}_{A}=k^{2}_{B}=k^{2}_{C(A)}=k^{2}_{AB}=0.01\theta^{2}$ and
$k_{E}=(1-\theta)^{2}$, it holds:

$E(F_{A})=\frac{6((1-\theta)^{2}+0.19\theta^{2})}{4((1-\theta)^{2}+0.03\theta^{2})+2((1-\theta)^{2}+0.04\theta^{2})}$
(36) $E(F_{B})=1+\frac{0.16\theta^{2}}{(1-\theta)^{2}+0.04\theta^{2}}$ (37)
$E(F_{C}(A))=1+\frac{0.03\theta^{2}}{(1-\theta)^{2}}$ (38)
$E(F_{AB})=1+\frac{0.04\theta^{2}}{(1-\theta)^{2}}$ (39)

Using Eqs. (36)-(39), we can plot the expected F-ratios for the set of values
of $\theta$, as illustrated in Figure 4(a), and compare them to the averaged
F-ratios obtained from the 1000 datasets simulated to compute the RPCs in
Figure 3. These averaged F-ratios are in Figure 4(b). We can see that the
theoretical and numerical results match perfectly, which shows that our
simulation approach accurately follows the ANOVA theory. We can also see that
the F-ratios alone cannot explain the relative power observed in the RPCs of
Figure 3: for instance, the F-ratio profiles of $A$ and $B$ are similar, while
the relative power in the RPCs are not.

(a) Expected F-ratio (b) Mean F-ratio

Figure 4:  Theoretical expected F-ratio (a) and mean simulated F-ratio (b) in
terms of the effect size for the RPC in Figure 3. Figure 5: Null distributions
for the first simulated dataset in the RPC in Figure 3 and for $\theta=0.5$.

The discrepancy between F-ratio and RPC profiles is caused by the different
null distributions of factors and interactions. We illustrate this reality in
Figure 5, which shows the null distributions of the factors and interaction
generated with permutation testing for the first of the 1000 simulated
datasets in the RPC and for $\theta=0.5$. The null distribution of $B$ is
significantly wider than the others. Since the p-value is obtained by
comparing the F-ratio to the null distribution, and $A$ and $B$ show similar
F-ratios at $\theta=0.5$ (Figure 4), the wider null distribution in $B$ makes
the power curve to rise slower than that in $A$ in Figure 3. The ASCA table
for the same simulated dataset is in Table 1. We can see in the table that the
F-ratios approximately follow Figure 4 and the p-values Figure 3. Note that
his dataset is a single instance of the distribution that is averaged in the
figures, which is the reason why some discrepancy is expected.

| SumSq | PercSumSq | df | MeanSq | F | Pvalue
---|---|---|---|---|---|---
Mean | 0.34652 | 2.7832 | 1 | 0.34652 | NaN | NaN
A | 0.86849 | 6.9757 | 3 | 0.2895 | 1.1096 | 0.006993
B | 0.546 | 4.3855 | 2 | 0.273 | 1.0613 | 0.16484
C(A) | 3.1529 | 25.3243 | 12 | 0.26274 | 1.0522 | 0.024975
AB | 1.5434 | 12.397 | 6 | 0.25724 | 1.0302 | 0.15984
Residuals | 5.9928 | 48.1342 | 24 | 0.2497 | NaN | NaN
Total | 12.4502 | 100 | 48 | 0.25938 | NaN | NaN
Table 1: ASCA table for the first simulated dataset in the RPC in Figure 3 and
for $\theta=0.5$.

Figure 6 presents several RPC examples similar to the first one, but where
some of the factors or the interaction are deactivated with a null standard
deviation. The figure shows that for any case where $k_{B}=0$, $k_{C(A)}=0$
and/or $k_{AB}=0$, the corresponding RPC ends up going to 0. This behaviour
can be explained as follows. From Eqs. (33), (34) and (35), we can see that if
we set the corresponding variance of the factor/interaction to 0, the expected
F-ratio equals 1 regardless the variance of the error. From Eq. (18) we can
see that as $\theta$ grows, the variance of the error tends to 0, decreasing
the expected type I error and taking the RPC to 0. This behaviour is not found
for factor A, and the reason can also be found in the corresponding equation
of the F-ratio, see Eq. (32). This equation represents an approximate test
rather than an exact one [9, 2]. If we set $k_{A}=0$, and adjust the remaining
population variances to the square of the standard deviation coefficients in
terms of $\theta$, it now holds:

$E(F_{A})=\frac{6((1-\theta)^{2}+0.07\theta^{2})}{4((1-\theta)^{2}+0.03\theta^{2})+2((1-\theta)^{2}+0.04\theta^{2})}$
(40)

The evolution of this expectation, together with the corresponding numerical
results, was also included in Figure 4 (marked with the label $A=0$), and for
comparison we also added the case where only $k_{B}=0$ (marked with the label
$B=0$). We can see that the approximate F-ratio in $A$ does not succeed in
cancelling out the F-ratio for a null variance of the factor, which can be
seen to lead to the misleading RPC of $A$ that shows an unrealistic power.
This undesirable behaviour in the RPC of factor $A$ remains even when the
variance of $C(A)$ or $AB$ is also cancelled (Figures 6(f) and 6(g),
respectively), and the RPC only works as expected when either $C(A)$ or $AB$
is not considered in the model, so that the F-ratio of $A$ corresponds to a
perfect test (Figure 6(h)).

(a) $k_{B}=k_{C(A)}=k_{AB}=0.1$, $k_{A}=0$ (b) $k_{A}=k_{C(A)}=k_{AB}=0.1$,
$k_{B}=0$

(c) $k_{A}=k_{B}=k_{AB}=0.1$, $k_{C(A)}=0$ (d) $k_{A}=k_{B}=k_{C(A)}=0.1$,
$k_{AB}=0$

(e) $k_{A}=k_{B}=0.1$, $k_{C(A)}=k_{AB}=0$ (f) $k_{B}=k_{AB}=0.1$,
$k_{A}=k_{C(A)}=0$

(g) $k_{B}=k_{C(A)}=0.1$, $k_{A}=k_{AB}=0$ (h) $k_{B}=k_{AB}=0.1$, $k_{A}=0$

Figure 6: Examples of Relative Population Curve for model (1). The design
matrix $\mathbf{F}$ contains a full factorial design with four levels in $A$,
three levels in $B$ and four individuals in each cell of $C(A)$. Other inputs
are $M=400$, $k_{E}=1$, $R=1000$, $P=200$, $\delta=0.1$ and $\alpha=0.05$.

###  6.2 Absolute Population Curves

Figure 7 shows four examples of APCs computed from the same parameters as the
RPC in Figure 3 and for $\theta=0.5$. The first example of APCs iteratively
replicates the whole experiment (Figure 7(a)) and the remaining examples
iteratively increase the number of levels of each of the factors (Figures
7(b), 7(c) and 7(d) for factors $A$, $B$ and $C(A)$, respectively). For
reference, we added the same baseline situations in all APCs and the original
RPC in Figure 3 marked with a red dashed line. Thus, the dashed line at
$\theta=0.5$ in Figure 3 identifies the same simulation point as at $\eta=1$
in Figure 7(a), $\eta=4$ (for four levels in $A$) in Figure 7(b), $\eta=3$
(for three levels in $B$) in Figure 7(c), and $\eta=4$ (for four replicates in
$C(A)$, $r_{C{A}}=4$) in Figure 7(d).

In general, we can see that increasing the replicates enhances the power in
all factors and the interaction for all APCs. As discussed in Figure 1, for a
fixed effect size ($\theta=0.5$), this enhancement is motivated by a reduction
in the variance of the null and the alternative distributions of the test. Let
us discuss this for each of the four examples.

(a) Whole experiment replicated (b) Factor $A$ replicated (c) Factor $B$
replicated (d) Factor $C(A)$ replicated

Figure 7: Examples of Absolute Population Curve for model (1). The original
design matrix $\mathbf{F}$ contains a full factorial design with four levels
in $A$, three levels in $B$ and four individuals in each cell of $C(A)$. Other
inputs are $M=400$, $k_{A}=k_{B}=k_{C(A)}=k_{AB}=0.1\theta$ and
$k_{E}=1-\theta$ for $\theta=0.5$, $R=1000$, $P=200$ and $\alpha=0.05$.

In Figure 7(a), we duplicate the whole experiment, but all levels of the
factors remain the same. This makes all variance coefficients in Eqs.
(32)-(35), with the exception of the variance of the error, to be multiplied
by a factor of 2 (and in general of $\eta$ if the experiment is duplicated
$\eta$-wise). This makes expected MSs, and in turn the expected F-ratios,
larger. The null distribution of a single instance simulated with this
duplication (Figure 8(a)) remains similar to the original one with no
duplication in Figure 5, with the exception of factor $A$ that shows a wider
shape. We can also compare the ASCA tables of the case with and without
duplication, Tables 2 and 1, respectively. As expected from the APC in Figure
7(a), after duplication all the factors and the interaction are statistically
significant.

Figure 8: Null distributions for the first simulated dataset in the RPC in
Figure 3 and for $\theta=0.5$, but when the Whole experiment replicated (a),
the number of levels of $A$ is duplicated (b), the number of levels of $B$ is
duplicated (c), and the number of levels of $C(A)$ is duplicated (d).

In Figure 7(b), the APCs show the behaviour of the power in terms of the
number of levels of $A$, $L_{A}$. From all the expected MSs values, only
$E(MS_{B})$ in Eq. (33) is affected by $L_{A}$. Consequently, the APC with
larger slope in terms of $L_{A}$ is actually $B$. All the other factors and
the interaction are increase their power with $L_{A}$, but rather than because
of a change in expected MS and/or F-tatio, they do because of a change in
their null distribution. This can be seen by comparing Figure 5, for
$L_{A}=4$, with Figure 8(b), for $L_{A}=8$. The later shows a clear reduction
in the variance of the null distributions of $C(A)$ and $AB$, and to a lower
extent of $A$. This reduction of variance leads to an increase of power.
Finally, comparing Tables 1 and 3, we can see that the duplication of the
levels of $A$ clearly reduces the p-value in $B$ and $C(A)$.

Figures 7(c) and 7(d) illustrate that the power in factor $C(A)$ is mostly
affected by the increase of levels of $B$, $L_{B}$, and the power of the
interaction $AB$ is mostly affected by the replicates in $C(A)$, $r_{C(A)}$.
All the other factors and interactions also increase their power. Again, this
increase of power is a complex mixture of a modification of the expected MSs
and F-ratios, and a reduction of the variance in the null distribution. The
latter is easily observed through numerical simulations (Figures 8(c) and 8(d)
and Tables 4 and 5).

As a general conclusion, we can see that sampling size in the form of
replicates and varying number of levels in the factors can have a complex
influence on the relative power of factors and their interactions. In complex
practical cases, an easy way to understanding how any form of duplication
affect the power of each given factor and/or interaction is through the APC
algorithm.

##  7 Conclusion

In this paper, we introduce the population power curves for ASCA and
demonstrate them in simulation, discussing their linkage with ANOVA theory and
deriving two useful forms of curve: Relative Population Curves (RPCs) and
Absolute Population Curves (APCs). RPCs are useful to find the optimal ASCA
pipeline for the analysis of an experimental design at hand. APCs are useful
to determine the sample size and the optimal number of levels for each factor
during the planning phase on an experiment. We believe that both tools should
be adopted by ASCA practitioners to plan their experimental design (APCs) and
analysis pipeline (RPCs).

## Software

All simulated examples in the paper can be reproduced with the code available
at https://github.com/CoDaSLab/PopulationCurvesASCA

## Acknowledgement

This work was supported by the Agencia Estatal de Investigación in Spain,
MCIN/AEI/ 10.13039/501100011033, grant No PID2020-113462RB-I00.

##  Appendix A Tables

ASCA tables for a single instance (dataset) simulate with the same parameters
of the RPC in Figure 3 and for $\theta=0.5$, but i) when the whole experiment
is duplicated (Table 2); ii) when the number of levels of $A$, $L_{A}$, is
duplicated (Table 3); iii) when the number of levels of $AB$, $L_{B}$, is
duplicated (Table 4); and iv) when the number of replicates in $C(A)$,
$r_{C(A)}$, is duplicated (Table 5);

| SumSq | PercSumSq | df | MeanSq | F | Pvalue
---|---|---|---|---|---|---
Mean | 0.43108 | 1.7257 | 1 | 0.43108 | NaN | NaN
A | 0.97207 | 3.8914 | 3 | 0.32402 | 1.1885 | 0.000999
B | 0.69985 | 2.8017 | 2 | 0.34992 | 1.2888 | 0.000999
C(A) | 3.2783 | 13.1239 | 12 | 0.27319 | 1.0946 | 0.000999
AB | 1.6291 | 6.5215 | 6 | 0.27151 | 1.0879 | 0.006993
Residuals | 17.9693 | 71.9358 | 72 | 0.24957 | NaN | NaN
Total | 24.9797 | 100 | 96 | 0.26021 | NaN | NaN
Table 2: ASCA table for the first simulated dataset in the RPC in Figure 3 and for $\theta=0.5$, when the whole experiment is duplicated.  | SumSq | PercSumSq | df | MeanSq | F | Pvalue
---|---|---|---|---|---|---
Mean | 0.42013 | 1.6809 | 1 | 0.42013 | NaN | NaN
A | 2.0342 | 8.1388 | 7 | 0.2906 | 1.1314 | 0.000999
B | 0.6513 | 2.6058 | 2 | 0.32565 | 1.2794 | 0.000999
C(A) | 6.1968 | 24.7931 | 24 | 0.2582 | 1.0219 | 0.12388
AB | 3.5635 | 14.2575 | 14 | 0.25454 | 1.0074 | 0.34965
Residuals | 12.1281 | 48.5239 | 48 | 0.25267 | NaN | NaN
Total | 24.994 | 100 | 96 | 0.26035 | NaN | NaN
Table 3: ASCA table for the first simulated dataset in the RPC in Figure 3 and for $\theta=0.5$, when the number of levels of $A$ is duplicated.  | SumSq | PercSumSq | df | MeanSq | F | Pvalue
---|---|---|---|---|---|---
Mean | 0.38426 | 1.5426 | 1 | 0.38426 | NaN | NaN
A | 0.98398 | 3.9501 | 3 | 0.32799 | 1.2495 | 0.000999
B | 1.4733 | 5.9146 | 5 | 0.29467 | 1.1578 | 0.000999
C(A) | 3.2698 | 13.1262 | 12 | 0.27248 | 1.0913 | 0.000999
AB | 3.8176 | 15.3253 | 15 | 0.25451 | 1.0193 | 0.17383
Residuals | 14.9814 | 60.1413 | 60 | 0.24969 | NaN | NaN
Total | 24.9103 | 100 | 96 | 0.25948 | NaN | NaN
Table 4: ASCA table for the first simulated dataset in the RPC in Figure 3 and for $\theta=0.5$, when the number of levels of $B$ is duplicated.  | SumSq | PercSumSq | df | MeanSq | F | Pvalue
---|---|---|---|---|---|---
Mean | 0.41662 | 1.6705 | 1 | 0.41662 | NaN | NaN
A | 1.0153 | 4.0712 | 3 | 0.33845 | 1.3083 | 0.000999
B | 0.66638 | 2.672 | 2 | 0.33319 | 1.2989 | 0.000999
C(A) | 7.2566 | 29.0969 | 28 | 0.25916 | 1.0333 | 0.01998
AB | 1.5392 | 6.1716 | 6 | 0.25653 | 1.0228 | 0.21978
Residuals | 14.0454 | 56.3178 | 56 | 0.25081 | NaN | NaN
Total | 24.9395 | 100 | 96 | 0.25979 | NaN | NaN
Table 5: ASCA table for the first simulated dataset in the RPC in Figure 3 and
for $\theta=0.5$, when the number of levels of $C(A)$ is duplicated.

## References

  * [1] A. K. Smilde, J. J. Jansen, H. C. Hoefsloot, R.-J. A. Lamers, J. Van Der Greef, and M. E. Timmerman, “Anova-simultaneous component analysis (asca): a new tool for analyzing designed metabolomics data,” Bioinformatics, vol. 21, no. 13, pp. 3043–3048, 2005.
  * [2] D. Montgomery, Design and Analysis of Experiments.  Wiley, 2020.
  * [3] M. Thiel, B. Feraud, and B. Govaerts, “ASCA+ and APCA+: extensions of ASCA and APCA in the analysis of unbalanced multifactorial designs,” Journal of Chemometrics, vol. 31, no. 6, p. e2895, 2017.
  * [4] M. Martin and B. Govaerts, “Limm-pca: Combining asca+ and linear mixed models to analyse high-dimensional designed data,” Journal of Chemometrics, vol. 34, no. 6, p. e3232, 2020.
  * [5] T. S. Madssen, G. F. Giskeødegård, A. K. Smilde, and J. A. Westerhuis, “Repeated measures asca+ for analysis of longitudinal intervention studies with multivariate outcome data,” PLoS Computational Biology, vol. 17, no. 11, p. e1009585, 2021.
  * [6] J. Camacho, C. Díaz, and P. Sánchez-Rovira, “Permutation tests for asca in multivariate longitudinal intervention studies,” Journal of Chemometrics, vol. 37, no. 7, p. e3398, 2023.
  * [7] A. M. G. C. C. J. Polushkina, O.A., “On missing data, outliers and transformations in permutation testing for asca and related factorizations,” In preparation, 2023.
  * [8] Z. Zhang and K.-H. Yuan, Practical Statistical Power Analysis Using Webpower and R.  05 2018.
  * [9] M. Anderson and C. T. Braak, “Permutation tests for multi-factorial analysis of variance,” Journal of statistical computation and simulation, vol. 73, no. 2, pp. 85–113, 2003.
  * [10] D. J. Vis, J. A. Westerhuis, A. K. Smilde, and J. van der Greef, “Statistical validation of megavariate effects in asca,” BMC bioinformatics, vol. 8, no. 1, pp. 1–8, 2007.
  * [11] F. Koleini, S. Hugelier, M. A. Lakeh, H. Abdollahi, J. Camacho, and P. J. Gemperline, “On the complementary nature of anova simultaneous component analysis (asca+) and tucker3 tensor decompositions on designed multi-way datasets,” Journal of Chemometrics, p. e3514, 2023.
  * [12] G. Zwanenburg, H. C. Hoefsloot, J. A. Westerhuis, J. J. Jansen, and A. K. Smilde, “ANOVA-Principal Component Analysis and ANOVA-Simultaneous Component Analysis: a comparison,” Journal of Chemometrics, vol. 25, no. 10, pp. 561–567, 2011.
  * [13] M. S. Armstrong and J. Camacho, “Chapter 4 - replicate analysis for uncertainty estimation in parafac and parafasca analyses of factorial metabolomics data,” in Fundamentals and Applications of Multiway Data Analysis (A. C. Olivieri, G. M. Escandar, H. C. Goicoechea, and A. M. de la Peña, eds.), vol. 33 of Data Handling in Science and Technology, pp. 61–81, Elsevier, 2024.
  * [14] M. A. Babyak, “What you see may not be what you get: a brief, nontechnical introduction to overfitting in regression-type models,” Psychosomatic medicine, vol. 66, no. 3, pp. 411–421, 2004.
  * [15] F. Marini, D. de Beer, E. Joubert, and B. Walczak, “Analysis of variance of designed chromatographic data sets: The analysis of variance-target projection approach,” Journal of Chromatography A, vol. 1405, pp. 94–102, 2015.