# Credibility of high $R^{2}$ in regression problems: a permutation approach

M. G. Ciszewskia and J. Söhla and A. J. R. Leenenb and B. van Trigtc and G.
Jongbloeda CONTACT M. G. Ciszewski. Email<EMAIL_ADDRESS>aApplied
Mathematics, Delft University of Technology, Mekelweg 4, 2628 CD Delft,
Netherlands;
bFaculty of Behavioural and Movement Sciences, VU University Amsterdam, Van
Der Boechorststraat 7, 1081 BT Amsterdam, Netherlands;
c Biomechanical Engineering, Delft University of Technology, Mekelweg 2, 2628
CD Delft, Netherlands

###### Abstract

The question of whether Y can be predicted based on X often arises and while a
well-adjusted model may perform well on observed data, the risk of overfitting
always exists, leading to poor generalization error on unseen data. This paper
proposes a rigorous permutation test to assess the credibility of high $R^{2}$
values in regression models, which can also be applied to any measure of
goodness of fit, without the need for sample splitting, by generating new
pairings of $(X_{i},Y_{j})$ and providing an overall interpretation of the
model’s accuracy. It introduces a new formulation of the null hypothesis and
justification for the test, which distinguishes it from previous literature.
The theoretical findings are applied to both simulated data and sensor data of
tennis serves in an experimental context. The simulation study underscores how
the available information affects the test, showing that the less informative
the predictors, the lower the probability of rejecting the null hypothesis,
and emphasizing that detecting weaker dependence between variables requires a
sufficient sample size.

###### keywords:

Permutation testing; model credibility; sensor data; $R^{2}$; regression
Word count (abstract): 169
Word count (without abstract, figures and references): 5329

##  1 Introduction

With the ubiquity of data often the question whether a response $Y$ can be
predicted based on predictors $X$ arises. A chosen model might perform well or
not on the observed data. If it does not perform well then one might try a
more complex model. No matter whether the initial model was fitting the
observed data well or whether a more complex model was chosen, a model well-
adjusted to the data always comes along with the risk of overfitting. Does the
model capture the essential structure or is it just fitted to the noise in the
observed data? Overfitting leads to poor generalization error on unseen data
and thus needs to be avoided.

In this paper we develop a method to test credibility of high values of
$R^{2}$ in regression models. Our method does not require sample splitting,
which is in contrast to the common approach of separating the data into a
train and a test set and also different from cross-validation, where the model
is fitted repeatedly only to parts of the available data. Our method is based
on recombining the pairings between predictors and responses through
permutations. In this way artificial reference datasets are created and the
performance of the model on the original data can then be assessed by
comparing it to the performances on the artificial reference datasets. The
purpose of our test is to ascertain whether the model is capable of fitting
the data more effectively than mere random noise.

There are several factors that may impede the feasibility of using a model to
predict $Y$ based on $X$. Overfitting is one such factor, however, inadequacy
of the model to detect the relationship between $X$ and $Y$ can also
contribute to this limitation. A trivial example of this could be applying a
linear regression model to predict $Y$ using $X$, when the ground truth
differs vastly from the adopted linear model, e.g. $Y=X^{2}+\epsilon$. In such
cases, augmenting the complexity of the model may be necessary to overcome
this challenge.

The main contribution of this paper is a rigorous formulation of a permutation
test for verifying the credibility of high $R^{2}$ in a fitted regression
model. Moreover, the test in its current form is not specific to $R^{2}$, but
is applicable to any measure of goodness of fit in regression analysis.
Because of its interpretability, $R^{2}$ is our test statistic of choice, but
this can be adapted if necessary. In contrast to cross-validation, the key
distinction of our approach is that instead of dividing the dataset into
separate training and testing subsets, we generate new pairings of
$(X_{i},Y_{j})$ conditional on the $X_{i}$ for $i=1,...,n$ and $Y_{j}$ for
$j=1,...,n$. The test statistic, as a measure of goodness of fit, provides an
overall interpretation of the model’s accuracy that extends beyond the
identification of overfitting, much like cross-validation. This paper
introduces a new formulation of the null hypothesis and provides a rigorous
justification for a permutation test that has been described in various forms
in the literature, for instance in its application in a subgroup discovery
problem in [5]. The main use case for this method is in the initial stages of
the model selection to test whether a given model can be applied effectively
to a dataset.

The outline of the paper is as follows. Section 2.1 formulates the problem and
introduces necessary notation. Section 2.2 gives a historical context and an
overview of the current state of the literature on permutation tests in the
context of regression problems. Section 2.3 contains the formal formulation of
the null hypothesis, theoretical considerations and the succinct description
of the method. Section 3.1 gives the results of the simulation study on the
permutation test in various scenarios regarding the set of predictors and
responses. Section 3.2 contains the application of the permutation test to
sensor data of tennis serves in order to describe the method accurately and
showcase its power in a real-life scenario.

##  2 Methodology

###  2.1 Problem description

Consider a regression setting. Given an observed pair $(X,Y)$, where $X$ is a
random vector and $Y$ is a real random variable. $Y$ is modelled as:

$Y=f(X)+\epsilon,$

where $f\in\mathcal{F}$ for some class of functions $\mathcal{F}$ and
$\epsilon$ is an error term. An example of $\mathcal{F}$ could be a set of all
linear functions of a fixed number of variables corresponding to a linear
regression model or a set of functions that can be described by a neural
network. Nonparametric classes of functions can also be considered, for
instance a set of log-concave functions. For the remainder of the paper, we
will focus on $R^{2}$ as goodness-of-fit measure.

Since the actual relationship between $X$ and $Y$ is not known in practice, a
chosen model $\mathcal{F}$ through which that relationship is described does
not need to be appropriate. The class of functions $\mathcal{F}$ is
misspecified if it does not contain $f$, while if it contains too many
functions, the model might be overfitting by memorizing the noise $\epsilon$.
In a real world scenario, we are often facing datasets that feature high-
dimensional, time-dependent or functional variables. The question whether
there is a relationship between $X$ and $Y$ and which model to choose to
describe it, is crucial. In this paper, we focus on the following aspect:

  * •

given a model $\mathcal{F}$, can we trust it if a high $R^{2}$ (or some other
goodness-of-fit measure) is found? Or is it because the model is able to fit
the noise?

Consider this simple example. Let $X_{1},X_{2}$ be independent standard normal
variables and $Y=X_{1}^{2}+X_{2}^{2}+\epsilon$, where
$\epsilon\sim\mathcal{N}(0,0.01)$. Consider a multi-layered neural net as a
model of choice to predict $Y$ using $X_{1}$ and $X_{2}$. For small sample
sizes shuffling the vector of responses and applying our model to this
shuffled dataset can yield values of $R^{2}$ higher than values of $R^{2}$
calculated for the model applied to the original dataset. Out of ten random
draws of a sample of size 10, five yielded higher values of $R^{2}$ for at
least one shuffled dataset than for the original pairing (we considered 200
shuffles of the sample). In applied settings, where the sample size is fixed
and difficult to increase, this presents an inherent issue. Sample size has an
immediate influence on the credibility of the model and needs to be taken into
account. This example illustrates how complex the issue of choosing a model
is.

A natural approach to address our problem is hypothesis testing. Our test is
permutation-based, thus we will give a brief review of the existing literature
on permutation tests, before we specify the precise null hypothesis.

###  2.2 Permutation tests in the existing literature

The idea of permutation tests existed before it was feasible to realize them
on larger scale and one of the main pioneers in that regard was Ronald Fisher.
Fisher’s interest in randomization as a concept can be traced back to his 1925
_Statistical Methods for Research Workers_ ([7]). His initial idea related to
the experimental design and ridding of the researcher’s bias into the setup of
experiments ([9]). At that time the idea to use randomization concerned only
experiments with smaller numbers of samples (as it would be near impossible to
apply it in case of larger sample size). The application of randomization to
hypothesis testing first emerged in the 1930s as introduced by Ronald Fisher
in the form of the permutation test as the test for comparing the
distributions of two independent groups. The use of the Fisher-Pitman
permutation test has since been extended to many other settings [1].

The rise of automated computing made these types of tests feasible to use on a
larger scale. Since the 1970s, permutation tests have started to enjoy larger
popularity. The permutation test has been further developed by [15], who
introduced a method for determining the p-value of the test. The paper [2]
discusses the growing popularity of the permutation test as a substitute for
the ANOVA F-test and investigate its robustness. An excellent assembly of
various papers in the biomedical field can be found in [14]. It showcases the
increased popularity of both the randomization in experimental design (as
opposed to random sampling) as well as the permutation tests for differences
in location. These types of papers have been seen also in other fields, e.g.
[1] in psychological research.

The applications of permutation tests have expanded beyond the comparison of
statistical models to a variety of problems. However, the formal formulation
of the null hypothesis can be challenging and subject to variations among
authors, particularly when permutation tests are utilized as a problem-solving
approach. It is our goal now to present examples of those null hypotheses in
different publications to provide context for our own formulation. In the two-
sample problem, [8] proposes a formulation of the null hypothesis that applies
the concept of exchangeability, i.e. the joint distribution of the
observations is invariant under permutations of the order of the predictors.
While their approach considers the linear regression model exclusively, it
does not reflect this consideration in the formulation of the null hypothesis.
The approach to two-sample (or $k$-sample) problems with null hypotheses
defined in terms of exchangeability is fairly common and well studied and can
be found in [4], [10] and [11]. The exploration of the robustness of
permutation tests without the assumption of exchangeability has been studied
as well, e.g. in [16] and [17]. A permutation test without the assumption of
exchangeability has been proposed more recently by [21]. Their test is
asymptotically exact and can be extended to testing hypothesis about one
population, a scenario that is also studied in this paper.

While linear regression models have received more interest in the context of
permutation tests, there have been applications to other models as well. A
permutation test for no effect in a functional linear regression model has
been proposed by [3]. In this case the randomization technique allows for the
simulation of the conditional distribution of the test statistic, which
otherwise would be difficult to obtain. In a subgroup discovery problem, [5]
employs a randomization technique. The idea is that the subgroups found by a
statistical model might contain false discoveries and we would like to detect
them. The test devised for this purpose is based on a null hypothesis that the
quality measure (for instance $R^{2}$) of a given subgroup is generated by the
distribution of false discoveries (DFD), which arises from the central limit
theorem applied to a set of quality measure values on some baseline subsets.
While the formulation is complicated, it corresponds to the idea of
permutation tests directly. The main issue with this approach lies in the fact
that the $p$-value calculated by the test is based on a single observation.
The lack of flexibility in permutation tests, particularly in the context of
experimental design of the model, is addressed by [20]. Their main focus is on
the tests of no effect in a general linear model and their main achievement is
the unification of a diverse set of results. The outcome is a single
permutation strategy with a single generalized statistic. It is worth noting
that, once again, the assumption of exchangeability is considered in the
formulation of the null hypothesis for the permutation test. An application to
weighted regression models can be found in [18]. The main problem with
weighted regression models is that the permutation of the response variables
and the permutation of the predictors do not lead to the same result. Hence,
the need for comparison of $X$-permutation and $Y$-permutation and their
variability in the weights.

Based on the publications mentioned in this section we can draw some
conclusions regarding the use of permutation tests and their advantages. The
main appeal of permutation tests stems from the fact that they do not require
any distributional assumptions on the population. The lack of assumption is
increasingly more interesting to researchers as the deep learning methods
become more popular since they do not rely on distributional assumptions.
Permutation tests are completely data-driven as pointed out by [1]. This can
be very appealing as the data is the main factor in shaping the distribution
of the test statistic, i.e. the test statistic can be chosen to be more easily
interpretable without focusing on its distribution. It is clear that
permutation tests arise in different settings and pertaining to different
problems. A part of the literature focuses on advancement of the theory of
permutation tests and another on using them in their own setting. The
application of permutation tests to address the issue of model trustability in
high-performing models has not been extensively studied in the literature.
Consequently, the subsequent section will concentrate on establishing a
rigorous theoretical foundation for this particular problem.

###  2.3 Permutation approach to testing the credibility of high $R^{2}$

Our goal is to investigate the credibility of $R^{2}$ values that result from
fitting $f(X)$ to $Y$ for $f\in\mathcal{F}$. We consider a test with null
hypothesis stated as follows:

$H_{0}:Y\textrm{ is independent of }f(X)\textrm{ for all }f\textrm{ in the
model }\mathcal{F}.$ (1)

$H_{0}$ represents the problem as described in section 2.1 well. If it were
true, then our model will not be able to capture the relation between $X$ and
$Y$ in a meaningful manner. In the context of the example from section 2.1,
considering a dataset with permuted responses will be no different to model
$\mathcal{F}$ under $H_{0}$. If $H_{0}$ is false, then the model $\mathcal{F}$
will be able to capture some aspects of the relation between $X$ and $Y$,
although it does not guarantee that the model is suitable and readily
applicable.

The null hypothesis $H_{0}$ as stated in (1) does not guide the choice of the
test statistic. In order to choose a suitable test statistic, further
understanding of $H_{0}$ is needed.

######  Proposition 2.1.

Let $(X_{1},Y_{1}),...,(X_{n},Y_{n})$ be an i.i.d. sample of $(X,Y)$. If
$f(X)$ and $Y$ are independent for all $f\in\mathcal{F}$, then the conditional
distribution of

$\\{(f(X_{i}),Y_{\tau(i)}):f\in\mathcal{F}\\}$ (2)

given the empirical measure $P_{n}^{X}$ of $X_{1},...,X_{n}$ and the empirical
measure $P_{n}^{Y}$ of $Y_{1},...,Y_{n}$ is the same for all permutations
$\tau$ of set $\\{1,...,n\\}$.

###### Proof.

For a given finite collection of functions
$f_{1},f_{2},...,f_{m}\in\mathcal{F}$ and a permutation $\tau$, the
conditional joint distribution of
$(f_{1}(X_{i}),Y_{\tau(i)}),...,(f_{m}(X_{i}),Y_{\tau(i)})$ given $P_{n}^{X}$
and $P_{n}^{Y}$ is the same as the joint distribution of

$(f_{1}(X_{i}),Y_{i}),...,(f_{m}(X_{i}),Y_{i}),$ (3)

thanks to the assumption of independence of $f(X)$ and $Y$ for any
$f\in\mathcal{F}$. Note that (3) is invariant with respect to the permutations
of $Y_{i}$. This statement will also be true if extended to a joint
distribution of (2) thanks to Kolmogorov extension theorem [13], hence the
distribution of joint conditional distribution of (2) given $P_{n}^{X}$ and
$P_{n}^{Y}$ is invariant with respect to the permutation of $Y_{i}$. ∎

Before proposition 2.1 is translated into a result in terms of $R^{2}$, we
formally define $R^{2}$. Consider $n$ realizations of $(X,Y)$ and denote them
as $(x_{1},y_{1}),...,(x_{n},y_{n})$. Let $L$ be a loss function and $\hat{f}$
be an empirical risk estimator in the sense that it minimizes

$\hat{f}=\operatorname*{arg\,min}_{f\in\mathcal{F}}\sum_{i=1}^{n}L(f(X_{i}),Y_{i}).$
(4)

Let $\hat{f}(x_{i})$ denote the prediction of $y_{i}$ for $i=1,...,n$. Then

$R^{2}=1-\frac{\sum_{i}(y_{i}-\hat{f}(x_{i}))^{2}}{\sum_{i}(y_{i}-\bar{y})^{2}},$
(5)

where $\bar{y}$ is the mean of the $y_{i}$. In the context of $R^{2}$,
proposition 2.1 implies the following result.

######  Proposition 2.2.

Let $(X_{1},Y_{1}),...,(X_{n},Y_{n})$ be an i.i.d. sample of $(X,Y)$. Assume
that $f(X)$ and $Y$ are independent for all $f\in\mathcal{F}$. Fix a
permutation $\tau$ of $\\{1,...,n\\}$ and a loss function $L$ defining an
empirical risk estimator as in (4). Then, conditionally on the empirical
measure $P_{n}^{X}$ of $X_{1},...,X_{n}$ and the empirical measure $P_{n}^{Y}$
of $Y_{1},...,Y_{n}$, the distribution of $R^{2}$ calculated based on data
$\\{(X_{i},Y_{\tau(i)})\\}$ using the aforementioned emprirical risk estimator
does not depend on $\tau$.

###### Proof.

Proposition 2.1 implies that the conditional distribution of

$\Bigg{\\{}\left(\sum\limits_{i=1}^{n}(Y_{\tau(i)}-f(X_{i}))^{2},\sum\limits_{i=1}^{n}L(f(X_{i}),Y_{\tau(i)})\right):f\in\mathcal{F}\Bigg{\\}}$
(6)

given $P_{n}^{X}$ and $P_{n}^{Y}$ is the same for all permutations $\tau$ of
set $\\{1,...,n\\}$. In combination with the definition (4) of $\hat{f}$ and
(5) of $R^{2}$, we conclude that for each permutation $\tau$, $R^{2}$
calculated for $\\{(X_{i},Y_{\tau(i)})\\}$ is sampled from the same
distribution conditioned on $P_{n}^{X}$ and $P_{n}^{Y}$ . ∎

This allows us to consider $R^{2}$ a viable choice for the test statistic.
Under the null hypothesis, the $R^{2}$ as calculated for $(x_{i},y_{i})$ is
sampled from the same distribution as the $R^{2}$ calculated for
$(x_{i},y_{\tau(i)})$ for some permutation $\tau$. The test itself is based on
permutations of the pairings $(x_{i},y_{i})$. We reject $H_{0}$ only if the
observed $R^{2}$ is much larger than ”most” of the $R^{2}$ obtained via random
permutations. Essentially we compare the observed $R^{2}$ to the distribution
of $R^{2}$ under $H_{0}$ given specific realizations of $X$ and $Y$, but not
their pairings. It is notable that $R^{2}$ can also be replaced by some other
statistic, as long as it can be calculated using the sample
$\\{(f(x_{i}),y_{i})\\}_{i}$. Proposition 2.1 permits other statistics to be
used instead of $R^{2}$. Taking $R^{2}$ as the test statistic is equivalent to
taking empirical risk with respect to quadratic loss as the test statistic. In
that sense, the other tests can also be constructed by considering empirical
risks with respect to other losses, e.g. absolute loss and Huber loss.

If the model $\mathcal{F}$ contains the constant functions, then $R^{2}$
calculated for a given $\mathcal{F}$ is always non-negative. This is true for
predictors which optimize with respect to the quadratic loss, since given set
of observations $\\{y_{i}\\}_{i=1,...,N}$, we can always choose
$f(X)\equiv\frac{1}{N}\sum\limits_{i=1}^{N}y_{i}$ which yields $R^{2}=0$. Note
that including the constants in $\mathcal{F}$, does not disturb the
independence of $Y$ and $f(X)$ for all $f\in\mathcal{F}$, since $Y$ is always
independent of a set of constant random variables. While $R^{2}$ is always
non-negative in linear regression models (if the intercept is included), that
is not the case for instance in the setting of neural nets.

The precise implementation of the test is as follows:

  * •

given original pairings of $(x_{i},y_{i})$, a model $\mathcal{F}$ and a chosen
prediction method, e.g. least squares, calculate the $R^{2}$, which we will
denote as $r_{0}^{2}$,111The specific method of prediction of $\hat{Y}_{i}$ is
not explicitly stated, but the approach is general.

  * •

choose a significance level $\alpha$ (default $\alpha=0.05$),

  * •

find the distribution of $R^{2}$ under the null hypothesis conditionally on
observed $x_{i}$ and $y_{(i)}$ for $i=1,...,n$ (approximated by the empirical
distribution function of $R^{2}$ values based on $\\{(x_{i},y_{\tau(i)}):1\leq
i\leq n\\}$, where $\tau$ are uniformly chosen permutations),

  * •

calculate the $1-\alpha$ quantile $q_{1-\alpha}$ of the empirical distribution
of $R^{2}$ values, and compare it to $r_{0}^{2}$,

  * •

if $r_{0}^{2}>q_{1-\alpha}$, then we reject the null hypothesis, otherwise we
do not reject it.

This implementation assumes that $R^{2}$ is the statistic of choice, but it
can be adapted to suit other statistics as well. The reason we prioritize
$R^{2}$ is primarily because of its benefits in terms of interpretability and
ease of use.

$R^{2}$ is bounded by 1 from above for any model. The proximity of $R^{2}$
values calculated from the permuted data or original $R^{2}$ values to 1 or to
each other can provide insight into goodness of fit of a model. The closer the
values of $R^{2}$ for the permuted data to 1, the greater the capability of
the model to fit to the noise. Close proximity of $q_{1-\alpha}$ to
$r_{0}^{2}$ in case of $r_{0}^{2}>q_{1-\alpha}$ and $r_{0}^{2}$ small implies
that the model’s predictive ability may not be satisfactory even though the
null hypothesis is rejected by the test. The test is widely applicable,
because of its general form and easily adaptable to different types of models.
It also provides an interesting commentary on the predictive abilities of a
chosen model. In the event that the quantile $q_{1-\alpha}$ for one model,
$\mathcal{F}_{1}$, significantly exceeds the same quantile for another model,
$\mathcal{F}_{2}$, we conclude that $\mathcal{F}_{1}$ is either overfitting,
indicating a need for reduction of the set of independent variables or model
simplification, or is better able to extract meaningful information from
unrelated data.

##  3 Application

###  3.1 Simulation study

We apply our permutation test in multiple scenarios. This section will
specifically focus on simulated datasets to assess the test’s performance on
datasets with varying dependence levels between $X$ and $Y$ and varying models
$\mathcal{F}$. An empirical example will be considered in section 3.2. In all
scenarios we consider the $R^{2}$-based test.

Two different models will be used throughout this section. One of them is a
linear regression model, which models the relationship between a random vector
$X$ and a random variable $Y$ in a linear manner: $Y=\beta\cdot X+\epsilon$.
The parameter $\beta$ will always be estimated using the least squares method.
Regardless of the length of vector $X$, this model will be referred to as
$\mathcal{F}_{\textrm{LR}}$. The other model of interest is a neural net. A
neural net is a collection of neurons arranged into layers, with neurons from
different layers connected to each other. Typically, a neural net consists of
an input layer, multiple hidden layers and an output layer. The estimation of
neural nets’ parameters, the weights associated with neurons and edges between
them, is done by feeding multiple training sets of inputs and outputs into the
net. Weights are adjusted each time based on a predefined cost function.
Neural nets will be referred to as $\mathcal{F}_{\textrm{NN}}$ with the number
of neurons on each layer specified as well as a $k$-tuple, where $k$ refers to
the number of layers, e.g. $\mathcal{F}_{\textrm{NN}}(30,30,30)$ is a neural
net with 3 hidden layers, each of which contains 30 neurons.

Let $X_{1},X_{2}\sim\mathcal{N}(0,1)$ and $Y\sim U([0,1])$ be independent
random variables. We consider two models: $\mathcal{F}_{\textrm{LR}}$ and
$\mathcal{F}_{\textrm{NN}}(30,30,30)$ and a sample of size $100$. In both
cases the null hypothesis is not rejected, see figures 1 and 2. In regards to
the models we chose, we can see that neural nets yield higher $R^{2}$ values
(in general). This is understandable as they are complex and more flexible
models and thus more prone to memorize the noise, which leads to overfitting.

Figure 1: Histogram of the distribution of generated $R^{2}$ using permutation
of $y$ values. The model considered here is linear regression
$\mathcal{F}_{\textrm{LR}}$. The sample size is 100. The red line denotes the
observed $R^{2}$ for the true pairings of $X$ and $Y$, the green line denotes
the 95%-quantile of the empirical distribution of $R^{2}$ (approximation using
200 permutations). Figure 2: Histogram of the distribution of generated
$R^{2}$ using permutation of $y$ values. The model considered here is a
3-layered neural net $\mathcal{F}_{\textrm{NN}}(30,30,30)$. The sample size is
100. The red line denotes the observed $R^{2}$ for the true pairings of $X$
and $Y$, the green line denotes the 95%-quantile of the empirical distribution
of $R^{2}$ (approximation using 200 permutations).

Now, let $X_{1}\sim\mathcal{N}(1,1),X_{2}\sim\mathcal{N}(0,1)$ be independent
and $Y=\log|X_{1}|+X_{2}^{2}+\epsilon$, where $\epsilon\sim\mathcal{N}(0,1)$
is the noise. Consider a sample of size $100$. In this case, the permutation
test rejects the null hypothesis, since the values of $R^{2}$ for the original
pairings are much higher than for any of the permuted pairings. Fig. 3 shows
the results of the test for the linear regression and fig. 4 shows the result
for the neural net. This particular example illustrates the test’s
applicability in the case of a functional relation between predictors and
responses. A linear regression model is yielding a rejection of $H_{0}$. This
shows the essence of this permutation test. The model is not just fitting the
noise, there is some relation between predictors and responses. It might not
be captured well using a linear regression model, but the model is still able
to capture more than pure noise.

Figure 3: Histogram of the distribution of generated $R^{2}$ using permutation
of $y$ values. The model considered here is linear regression
$\mathcal{F}_{\textrm{LR}}$. The sample size is 100. The red line denotes the
observed $R^{2}$ for the true pairings of $X$ and $Y$, the green line denotes
the 95%-quantile of the empirical distribution of $R^{2}$ (approximation using
200 permutations). Figure 4: Histogram of the distribution of generated
$R^{2}$ using permutation of $y$ values. The model considered here is a
3-layered neural net $\mathcal{F}_{\textrm{NN}}(30,30,30)$. The sample size is
100. The red line denotes the observed $R^{2}$ for the true pairings of $X$
and $Y$, the green line denotes the 95%-quantile of the empirical distribution
of $R^{2}$ (approximation using 200 permutations).

For the remaining scenarios, we consider only the linear regression model
$\mathcal{F}_{\textrm{LR}}$. We inspect the influence of changing the
distribution slightly on the test in order to ensure the statistical analysis
using the test is reliable and accurate. For $a\in\mathbb{R}$ let
$X_{1}\sim\mathcal{N}(a,1),X_{2}\sim\mathcal{N}(0,0.1)$ be independent and
$Y=\log|X_{1}|+X_{2}^{2}+\epsilon$, where $\epsilon\sim\mathcal{N}(0,0.1)$ is
the noise. Consider a sample of size $100$. Note that the variance of $X_{2}$
has been decreased in comparison to the previous example. Only for values of
$a$ close to 0, the null hypothesis is not rejected (fig 5). This makes sense,
since the logarithm changes most rapidly close to $0$ and for those arguments
it is difficult to fit in a linear function which describes this relationship
well. When the variance of $X_{2}$ increases to 0.5, the null hypothesis is no
longer rejected for some values of $a$ larger than 5 (fig 6). This particular
case shows the influence of available information on rejecting the null
hypothesis. The less informative predictors are the more likely it is not to
reject the null hypothesis; we can see that as the parameter $a$ increases,
the $\log|X_{1}|$ becomes flatter slowly losing its predictive value.

Figure 5: The plot shows the results of performing the permutation test for
linear regression model $\mathcal{F}_{\textrm{LR}}$. The sample size is 100.
The blue dots show the observed $R^{2}$ and the orange dots show the 95%
quantile of the empirical distribution of generated $R^{2}$ (approximation
using 200 permutations). The test has been performed for values of $a$ ranging
between 0 and 10. Figure 6: The plot shows the results of performing the
permutation test for linear regression model $\mathcal{F}_{\textrm{LR}}$. The
sample size is 100. The blue dots show the observed $R^{2}$ and the orange
dots show the 95% quantile of the empirical distribution of generated $R^{2}$
(approximation using 200 permutations). The test has been performed for values
of $a$ ranging between 0 and 10.

Using the bivariate normal distribution with varying correlation, we can
empirically detect the point at which the test rejects $H_{0}$ for the linear
regression model $\mathcal{F}_{\textrm{LR}}$ as the variables become more and
more dependent. Let $0\leq\epsilon\leq 1$ and $X,Y\sim N(\mu,\Sigma)$, such
that

$\mu=\begin{bmatrix}0\\\ 0\end{bmatrix},\Sigma=\begin{bmatrix}1&\epsilon\\\
\epsilon&1\end{bmatrix}.$

Figure 7 shows that as the correlation reaches 0.3, the test starts to reject
$H_{0}$ almost always in case of sample size $n=100$. Hence, the conclusion
here can be that for sample size $n=100$, the dependence is only detectable
reliably by the test when the correlation between variables is greater than
0.3. This particular example shows that for a given sample size a certain
threshold of correlation exists at which the test starts to reject the null
hypothesis. As the correlation increases the rejection becomes more and more
likely for a given sample size.

Figure 7: The plot shows the results of performing the permutation test for
linear regression model $\mathcal{F}_{\textrm{LR}}$. The sample size is 100.
The blue dots show the observed $R^{2}$ and the orange dots show the 95%
quantile of the empirical distribution of generated $R^{2}$ (approximation
using 200 permutations). The test has been performed for values of correlation
$\epsilon$ ranging between 0 and 1.

Lastly, figures 8 and 9 show explicitly the influence of the sample size on
the test’s capability to reject $H_{0}$ for the linear regression model
$\mathcal{F}_{\textrm{LR}}$. In the case when $H_{0}$ is true (fig. 8), the
null hypothesis is only rejected in a few cases for very small sample sizes.
In the case when $H_{0}$ is false (fig. 9), specifically with
$Y=\log(X)+\epsilon$ for $\epsilon\sim\mathcal{N}(0,1)$, the null hypothesis
is not rejected for smaller sample sizes (smaller than 200) and the rejection
rate increases as the sample size increases. We can conclude that in order to
detect less prominent dependence between variables, a sufficient sample size
is required.

Figure 8: The plot shows the results of performing the permutation test for
linear regression model $\mathcal{F}_{\textrm{LR}}$. The blue dots show the
observed $R^{2}$ and the orange dots show the 95% quantile of the empirical
distribution of generated $R^{2}$ (approximation using 200 permutations). The
test has been performed for sample sizes ranging between 10 and 1000. Figure
9: The plot shows the results of performing the permutation test for linear
regression model $\mathcal{F}_{\textrm{LR}}$. The blue dots show the observed
$R^{2}$ and the orange dots show the 95% quantile of the empirical
distribution of generated $R^{2}$ (approximation using 200 permutations). The
test has been performed for sample sizes ranging between 10 and 1000.

###  3.2 Tennis serve dataset

This section concerns an application of the permutation test to a tennis serve
dataset. Seven professional athletes wearing inertial measurement units (IMUs)
performed tennis serves. Each athlete followed a protocol of first and second
serves. Sensors were placed on 4 body parts: lower and upper arms, trunk and
pelvis as can be seen in figure 10. Each IMU contained a triaxial
accelerometer and triaxial gyroscope. The data consists of 7 uninterrupted
time series of 24-dimensional data (4 body parts $\times$ 2 types of sensors
$\times$ 3 axes). For more details regarding this dataset see [6].

Figure 10: Segment model of right-handed player and racquet (back view,
frontal plane).

Additionally, a dataset containing personal characteristics of the players and
performance characteristics of each serve has been included. Personal
characteristics refer to the sex, age, height and weight of the players.
Performance characteristics are the ball velocity, an indication of whether
the ball went in or out and the velocity-accuracy index (VA index). The VA
index for a single serve was introduced in [12] and is defined as follows:

$\textrm{VA index}=\frac{(\textrm{ball velocity
(kph)})^{2}}{100}\times\frac{\textrm{achieved points}}{9},$ (7)

where achieved points refer to the number of points assigned to a serve based
on its closeness to a target area on the court (see fig. 11). The number of
points assigned to a serve is based on a new Serve Tennis Test (STT)
introduced in [19]. The points are discrete. Nine points are given for hitting
the center of the target area. Six and three points are assigned for areas
around the center. One point is assigned for a ball further from the target
area, but still a valid ball, while zero points are given to a serve which did
go out. Each participant performed approximately 48 serves. In total, 29.6% of
serves were faults (and as a result had a VA-index 0).

Figure 11: Target areas for the tennis serve. The scenario considered here is
a serve in wide direction. The points given on each target area correspond to
the number of accuracy points needed to calculate the VA index of the serve.

The initial goal of analyzing the tennis serve dataset was to provide a good
prediction for the VA-index of each serve. However, we were not able to find
an appropriate model to provide a good prediction. In some limited dataset
scenarios, the neural net model $\mathcal{F}_{\textrm{NN}}(60,60,60)$ for VA-
index prediction was able to achieve sufficiently high values of $R^{2}$. The
size of limited datasets questioned the believability of such results. Hence,
our $R^{2}$-based permutation test was employed.

The functional predictors have been transformed into vectors, using a Fourier
basis representation, in order to be able to use the neural net
$\mathcal{F}_{\textrm{NN}}(60,60,60)$ and the linear regression model
$\mathcal{F}_{\textrm{LR}}$. The choice to use Fourier coefficients as
predictors was the most natural way of incorporating information from the time
series. First, a prediction of ball speed was considered. The permutation test
rejected the null hypothesis in cases of both models as seen in figures 12 and
13. The test rejects the null hypothesis for both models, although higher
values of $R^{2}$ achieved by the neural net for the original pairings suggest
greater capabilities of that model to detect the dependence.

Figure 12: Histogram of the distribution of generated $R^{2}$ using
permutation of $y$ values. The sample size is 46. The red line denotes the
observed $R^{2}$ for the true pairings of $X$ and $Y$, the green line denotes
the 95%-quantile of the empirical distribution of $R^{2}$ (approximation using
200 permutations).

5-fold cross-validation was performed for the linear regression model
$\mathcal{F}_{\textrm{LR}}$. The cross-validation is used here to improve the
prediction by selecting the best set of predictors. The mean test score222The
$R^{2}$ on the unseen data averaged over 5 splits. for the best predictor set
was equal to 0.43 with the mean train score 0.57. Cross-validation confirms
that the linear regression model can be applied to this dataset, although the
results are only decent. Applying a train-test split to the neural net yields
much higher test score values even without searching for the best predictor
set.

Figure 13: Histogram of the distribution of generated $R^{2}$ using
permutation of $y$ values. The sample size is 46. The red line denotes the
observed $R^{2}$ for the true pairings of $X$ and $Y$, the green line denotes
the 95%-quantile of the empirical distribution of $R^{2}$ (approximation using
200 permutations).

In the case of prediction of the VA-index defined in (7), the permutation test
did not reject the null hypothesis for the linear regression model
$\mathcal{F}_{\textrm{LR}}$ as well as for the neural net model
$\mathcal{F}_{\textrm{NN}}(60,60,60)$. Figure 14 shows that the initial
$R^{2}$ can be seen as promising, however, the test shows that for other
permutations of $y$ values, the model yields even higher $R^{2}$ values.
Figure 15 shows less promising results for the linear regression model. These
results convince us that a good prediction using the linear regression model
or the neural net model with the chosen structure is not possible at the
moment. The issue may lie with the current size of the dataset or the number
of serves per player or simply because the relation as can be described by the
neural net is not strong. The fact that the number of Fourier coefficients
used in this prediction was increased to achieve more favourable $R^{2}$ for
the original pairings of $(x_{i},y_{i})$ (at least in the case of the deep
learning model), shows how complex this task is and additional information is
needed in the data to increase the $R^{2}$.

Figure 14: Histogram of the distribution of generated $R^{2}$ using
permutation of $y$ values. The sample size is 34. The red line denotes the
observed $R^{2}$ for the true pairings of $X$ and $Y$, the green line denotes
the 95%-quantile of the empirical distribution of $R^{2}$ (approximation using
200 permutations). Figure 15: Histogram of the distribution of generated
$R^{2}$ using permutation of $y$ values. The sample size is 34. The red line
denotes the observed $R^{2}$ for the true pairings of $X$ and $Y$, the green
line denotes the 95%-quantile of the empirical distribution of $R^{2}$
(approximation using 200 permutations).

##  4 Conclusion and discussion

This paper concerns the theoretical foundations and the application of the
permutation approach to testing the credibility of high $R^{2}$ or of other
measures of goodness of fit. The test is a tool to determine whether a model
is able to fit the data better than pure noise. The test can serve as an
initial tool to determine the capability of given models to predict new data.
As such it can be used in combination with cross-validation, which is
primarily used for model selection, parameter tuning and estimating the out-
of-sample performance. The permutation test proposed in this paper does not
require a train/test split. The simulation study shows the performance of the
test on various different dependence scenarios. It also shows the influence of
the sample size on the test. The application to the tennis serve dataset shows
applicability in a real-life scenario. In this case, it gave evidence that a
well-fitting model is not necessarily trustworthy. The prediction is either
not possible with the given sensor data and model or a larger sample size is
needed to predict the VA-index more accurately.

### Declaration of interests

The authors declare that they have no known competing financial interests or
personal relationships that could have appeared to influence the work reported
in this paper.

### Code availability

Custom code for simulation study can be found at:
https://github.com/mgciszewski/credibility_2023.

### Data availability

The data that support the findings of this study are available from the
corresponding author upon request.

### Funding

This work is part of the research programme CAS with project number P16-28
project 2, which is (partly)

financed by the Dutch Research Council (NWO).

## References

  * [1] K.J. Berry, P.W. Mielke Jr., and H.W. Mielke, _The Fisher-Pitman permutation test: an attractive alternative to the $F$ test_, Psychol. Rep. 90 (2002), pp. 495–502. Available at https://doi.org/10.2466/pr0.2002.90.2.495.
  * [2] R.J. Boik, _The Fisher-Pitman permutation test: A non-robust alternative to the normal theory $F$ test when variances are heterogeneous_, Br. J. Math. Stat. Psychol. 40 (1987), pp. 26–42. Available at https://doi.org/10.1111/j.2044-8317.1987.tb00865.x.
  * [3] H. Cardot, A. Goia, and P. Sarda, _Testing for no effect in functional linear regression models, some computational approaches_ , Commun. Stat. Simul. Comput. 33 (2004), pp. 179–199. Available at https://doi.org/10.1081/SAC-120028440.
  * [4] D. Commenges, _Transformations which preserve exchangeability and application to permutation tests_ , J. Nonparametr. Stat. 15 (2003), pp. 171–185. Available at https://doi.org/10.1080/1048525031000089310.
  * [5] W. Duivesteijn and A. Knobbe, _Exploiting false discoveries - statistical validation of patterns and quality measures in subgroup discovery_ , in _Proc. 2011 IEEE 11th Int. Conf. Data Min._ , Vancouver, BC, Canada. IEEE, 2011, pp. 151–160.
  * [6] E. Faneker, _The kinetic chain and serve performance in elite tennis players (Master Research Project, Vrije Universiteit Amsterdam)_ , 2021\.
  * [7] R.A. Fisher, _Statistical methods for research workers_ , 1st ed., Oliver & Boyd, Edinburgh, Scotland, 1925.
  * [8] P.I. Good, _Extensions of the concept of exchangeability and their applications_ , J. Mod. Appl. Stat. Methods 1 (2002), pp. 243–247. Available at https://doi.org/10.22237/jmasm/1036110240.
  * [9] N.S. Hall, _R. A. Fisher and his advocacy of randomization_ , J. Hist. Biol. 40 (2007), pp. 295–325. Available at https://doi.org/10.1007/s10739-006-9119-z.
  * [10] Y. Huang, H. Xu, V. Calian, and J.C. Hsu, _To permute or not to permute_ , Bioinform. 22 (2006), pp. 2244–2248. Available at https://doi.org/10.1093/bioinformatics/btl383.
  * [11] A.D. Hutson and G.E. Wilding, _Maintaining the exchangeability assumption for a two-group permutation test in the non-randomized setting_ , J. Appl. Stat. 39 (2012), pp. 1593–1603. Available at https://doi.org/10.1080/02664763.2012.661707.
  * [12] N. Kolman, B. Huijgen, T. Kramer, M. Elferink-Gemser, and C. Visscher, _The Dutch Technical-Tactical Tennis Test (D4T) for talent identification and development: psychometric characteristics_ , J. Hum. Kinet. 30 (2017), pp. 127–138. Available at https://doi.org/10.1515/hukin-2017-0012.
  * [13] A.N. Kolmogorov, _Foundations of the theory of probability_ , 2nd ed., Chelsea Publishing Company, New York, 1956.
  * [14] J. Ludbrook and H. Dudley, _Why permutation tests are superior to $t$ and $F$ tests in biomedical research_, Am. Stat. 52 (1998), pp. 127–132. Available at https://doi.org/10.2307/2685470.
  * [15] A. Oden and H. Wedel, _Arguments for Fisher’s permutation test_ , Ann. Statist. 3 (1975), pp. 518–520. Available at https://doi.org/10.1214/aos/1176343082.
  * [16] J.P. Romano, _On the behavior of randomization tests without a group invariance assumption_ , J. Am. Stat. Assoc. 85 (1990), pp. 686–692. Available at https://doi.org/10.2307/2290003.
  * [17] R.L. Schmoyer, _Permutation tests for correlation in regression errors_ , J. Am. Stat. Assoc. 89 (1994), pp. 1507–1516. Available at https://doi.org/10.2307/2291013.
  * [18] C.J.F. ter Braak, _Predictor versus response permutation for significance testing in weighted regression and redundancy analysis_ , J. Stat. Comp. Simul. 92 (2021), pp. 2041–2059. Available at https://doi.org/10.1080/00949655.2021.2019256.
  * [19] T. van Dijk, T.A. Leenen, B. van Trigt, A. Hoekstra, and M.M. Hoozemans, _Development, construct validity and test-retest reliability of a Serve Tennis Test (STT) in elite tennis_ , (unpublished manuscript) .
  * [20] A.M. Winkler, G.R. Ridgway, M.A. Webster, S.M. Smith, and T.E. Nichols, _Permutation inference for the general linear model_ , Neuroimage 92 (2014), pp. 381–397. Available at https://doi.org/10.1016/j.neuroimage.2014.01.060.
  * [21] S. Zhang, _The split sample permutation $t$-tests_, Neuroimage 139 (2009), pp. 3512–3524. Available at https://doi.org/10.1016/j.jspi.2009.04.004.