Daniel J. SchadShravan Vasishth University of Potsdam, Potsdam,
Germany.University of Potsdam, Potsdam, Germany. Schad & Vasishth Please send
correspondence to<EMAIL_ADDRESS>
# The posterior probability of a null hypothesis given a statistically
significant result

###### Abstract

When researchers carry out a null hypothesis significance test, it is tempting
to assume that a statistically significant result lowers Prob(H0), the
probability of the null hypothesis being true. Technically, such a statement
is meaningless for various reasons: e.g., the null hypothesis does not have a
probability associated with it. However, it is possible to relax certain
assumptions to compute the posterior probability Prob(H0) under repeated
sampling. We show in a step-by-step guide that the intuitively appealing
belief, that Prob(H0) is low when significant results have been obtained under
repeated sampling, is in general incorrect and depends greatly on: (a) the
prior probability of the null being true; (b) Type I error, and (c) Type II
error. Through step-by-step simulations using open-source code in the R System
of Statistical Computing, we show that uncertainty about the null hypothesis
being true often remains high despite a significant result. To help the reader
develop intuitions about this common misconception, we provide a Shiny app
(https://danielschad.shinyapps.io/probnull/). We expect that this tutorial
will be widely useful for students and researchers in many fields of
psychological science and beyond, and will help researchers better understand
and judge results from null hypothesis significance tests.

###### keywords:

Null hypothesis significance testing, Bayesian inference, statistical power

††journal: Manuscript††volume:

Draft of

Null hypothesis significance testing (NHST), as it is practised in all areas
of science, involves a fairly straightforward procedure. We begin by positing
a null hypothesis $H_{0}$, usually a point null hypothesis that a parameter
$\mu$ has a specific value: $H_{0}:\mu=\mu_{0}$. Then we collect data, compute
the sample mean $\bar{x}$, and estimate the standard error $SE$ from the
sample standard deviation $s$ and sample size $n$ by computing
$SE=s/\sqrt{n}$. Next, we compute some statistic such as the observed
t-statistic, $t_{observed}=\frac{\bar{x}-\mu_{0}}{SE}$. If the absolute value
of the observed t-statistic is larger than the absolute value of some critical
t-value, we reject the null hypothesis. Usually we also compute the p-value,
which is the probability of obtaining the observed t-statistic, or a value
more extreme, assuming that the null hypothesis is true. Conventionally, when
the p-value is less than $0.05$, we reject the null hypothesis. A common
phrasing is to say that we have a “statistically significant” result, and that
the effect of interest is “reliable.” As is well known, an issue that is of
great importance here is false and true discovery rates (Betancourt, 2018).

The false discovery rate is the probability of incorrectly rejecting the null
when the null is in fact true; this is referred to as Type I error. It is
conventionally fixed at $0.05$. The true discovery rate is the probability of
correctly rejecting the null when $\mu$ has some specific point value that is
not the null value $\mu_{0}$; this is usually called power. The quantity
(1-power) is called Type II error, often written as $\beta$; it is the
probability of incorrectly accepting the null when it is false with some
specific value for $\mu$, i.e., when the true $\mu$ is some specific point
value other than $\mu_{0}$.

It is well-known that power needs to be high in order to draw inferences
correctly from a statistically significant result; when power is low,
statistically significant results are guaranteed to be overestimates or even
to have the wrong sign (Gelman Carlin, 2014), and they are likely to be
unreplicable (Vasishth ., 2018). In other words, a significant result under
low power is never “reliable” in any sense of the word.

Researchers sometimes assume that a significant result reduces the probability
of the null hypothesis being true to a low value. For example, a survey by Tam
. (2018) reports this widespread misunderstanding of the p-value by medical
doctors. As they put it:

> Many respondents conceptualised the P value as numerically indicating the
> natural probability of some phenomenon — for instance, a 95% or 5% chance of
> the truth or falsity of a hypothesis in the real world.

A second example comes from Doherty . (2002). On page 376, Table 2, they state
that “[the p-value] is the probability that the null hypothesis is true.” A
third example comes from a textbook written for medical researchers (Harris
Taylor, 2003). On their page 24, they write: “The P value is used when we wish
to see how likely it is that a hypothesis is true”; and on page 26, they
write: “The P value gives the probability that the null hypothesis is true.”

Here, we investigate the posterior probability of the null hypothesis being
true under different possible assumptions. Specifically, when power is low,
medium, or high, and when Type I error is 0.05, 0.01, or 0.005.

Intuitively, it does seem obvious that rejecting the null hypothesis after
finding a significant result leads us to belief that the posterior probability
of the null hypothesis being true is low. We will show in this paper that this
intuitive belief is in general wrong, except in two extreme situations that
are rarely or never realized: when statistical power is high (greater than
0.80) and Type I error is low (0.005).

Note that, technically, talking about “the probability that the null
hypothesis is true” is meaningless in the NHST framework. Probability mass
functions can only be associated with discrete outcomes that constitute a
random variable. An example from everyday life would be the probability of
catching a train when running late: there are two possible outcomes, either
one gets the train or not, and each outcome has a probability associated with
it. By contrast, the null hypothesis is not a random variable and therefore
cannot have a probability associated with the two possible outcomes of being
true or false. The null hypothesis is either true or it is false.

However, in order to talk about the probability of the null hypothesis being
true or false, we can assume for the moment that the null hypothesis is a
random variable. Suppose that before running the experiment, we begin with the
assumption that the null hypothesis is believed to be true with some
probability $\theta_{prior}$. Once we get a significant result, the
probability of the null being true should (intuitively) fall to some low value
$\theta_{posterior}$.

But these point values $\theta_{prior}$ and $\theta_{posterior}$ only partly
characterize our beliefs. Before running the experiment, we surely have some
uncertainty about the probability $\theta_{prior}$ that the null is true. For
example, the null may be true at the outset with probability somewhere between
85 and 95%. Thus, the prior probability of the null being true has a
probability distribution associated with it, it cannot be just a point value.
Still relying on intuition, we might say that a statistically significant
result should shift this probability distribution to some low range, say
10-20%. Such a hypothetical situation is visualized in Figure 1a.

Figure 1: Prior and posterior probability of the null hypothesis. a) An
illustration of how our belief in the null hypothesis—expressed as a
probability distribution— might hypothetically shift once we see a
statistically significant result. The vertical lines show the 95% credible
intervals. b) Prior probability of the null hypothesis being true, expressed
as a Beta(60,6) distribution.

To summarize, strictly speaking, the null hypothesis is either true or false,
it has no probability distribution associated with it. So, one cannot even
talk about the probability of the null hypothesis being true. Nevertheless, it
is possible to relax this strict stipulation and ask ourselves: how strongly
do we believe that the null is true before and after we do a significance
test? A domain expert working in a particular field should be able to state,
as a probability distribution, his or her a priori confidence level in a
particular null hypothesis. In practice, some expert elicitation might be
required (O’Hagan ., 2006; Oakley O’Hagan, 2010).

Bayes’ rule allows us to calculate this posterior probability of the null
hypothesis being true. Bayes’ rule states that, given a vector of data $y$, we
can derive the probability density function of a parameter or a vector of
parameters $\theta$ given data, $f(\theta\mid y)$, by multiplying the
likelihood function of the data, $f(y\mid\theta)$, with the prior probability
of the parameter(s), $f(\theta)$, and dividing by the marginal likelihood of
the data, $f(y)$:

$f(\theta\mid y)=\frac{f(y\mid\theta)f(\theta)}{f(y)}$ (1)

The marginal likelihood of the data can be computed by integrating out the
parameter(s) $\theta$:

$f(y)=\int f(x,\theta)\,d\theta=\int f(y\mid\theta)f(\theta)\,d\theta$ (2)

As McElreath (2016) mentions, Bayes’ rule can be used to work out the
posterior probability of the null being true given a significant effect under
repeated sampling. This is what we turn to next. Before we can carry out this
computation, we have to decide on the prior probability of the null hypothesis
being true; this is not too difficult to determine for specific research
questions. We could start by eliciting from a researcher their prior belief
about the probability of some particular null hypothesis being true:
$Prob(H_{0}\leavevmode\nobreak\ true)$. Given such a prior probability for the
null hypothesis, we then stipulate a Type I error,
$Prob(sig|H_{0}\leavevmode\nobreak\ true)=\alpha$ and a Type II error,
$Prob(not\leavevmode\nobreak\ sig|H_{0}\leavevmode\nobreak\ false)=\beta$.
(When we write $H_{0}\leavevmode\nobreak\ false$, we mean that the null is
false with some specific value for the parameter $\mu$). Once we have these
numbers, Bayes’ rule allows us to compute $Prob(H_{0}\leavevmode\nobreak\
true|sig)$, the posterior probability of the null being true given a
significant result under repeated sampling. In other words, Bayes’ rule helps
us quantify the extent to which our prior belief should shift in the light of
the long-run probability of obtaining a statistically significant result:

$Prob(H_{0}\leavevmode\nobreak\
true|sig)=\frac{Prob(sig|H_{0}\leavevmode\nobreak\ true)\times
Prob(H_{0}\leavevmode\nobreak\ true)}{Prob(sig)}$ (3)

The denominator $Prob(sig)$ is the marginal probability of obtaining a
significant effect under repeated sampling, and is straightforward to compute
using the law of total probability (Kolmogorov, 2018). This law states that
the probability of a random variable Z, $Prob(Z)$, given another random
variable A, is $Prob(Z|A)Prob(A)+Prob(Z|\neg A)Prob(\neg A)$. Translating this
to our particular question, the event $Z$ is the significant effect we
obtained under repeated sampling, and the event $A$ is the null hypothesis
being true or false.

$\begin{split}Prob(sig)=&Prob(sig|H_{0}\leavevmode\nobreak\
true)Prob(H_{0}\leavevmode\nobreak\ true)+Prob(sig|H_{0}\leavevmode\nobreak\
false)Prob(H_{0}\leavevmode\nobreak\ false)\\\ =&\alpha\times
Prob(H_{0}\leavevmode\nobreak\
true)+(1-\beta)\times(1-Prob(H_{0}\leavevmode\nobreak\ true))\\\
=&\alpha\times\theta+(1-\beta)\times(1-\theta)\\\ \end{split}$ (4)

The last line above arises because $Prob(sig|H_{0}\leavevmode\nobreak\
true)=\alpha$ (Type I error), $Prob(sig|H_{0}false)=1-\beta$ (power), and
$Prob(H_{0}\leavevmode\nobreak\ true)=\theta$ (the prior probability of the
null being true).

Thus, $Prob(H_{0}\leavevmode\nobreak\ true|sig)$ is really a function of three
quantities:

  1. 1.

The false discovery rate, or Type I error $\alpha$.

  2. 2.

The true discovery rate, or power $(1-\beta)$, where $\beta$ is Type II error.

  3. 3.

The prior probability of null being true ($\theta$).

We will now compute, under different assumptions, the posterior probability of
the null being true given a significant effect. Before we can do this, we have
to decide on a prior probability of the null hypothesis being true. What is a
reasonable prior distribution to start with? In a recent paper, Benjamin .
(2018) write the following: “Prediction markets and analyses of replication
results both suggest that for psychology experiments, the prior odds of H1
relative to H0 may be only about 1:10. A similar number has been suggested in
cancer clinical trials, and the number is likely to be much lower in
preclinical biomedical research.” A prior odds of 1:10 of the alternative
being true relative to the null means that the probability of the null being
true is about 90%. We take this estimate as a starting point; below we will
also consider alternative scenarios where the probability of the null being
true is lower.

For now, we will assume that the null hypothesis $H_{0}$ has the high prior
probability of 90% of being true. Just as a coin has heads and tails as
possible outcomes, the null hypothesis can have two possible outcomes, true or
false, each with some probability. Thus, we can now talk about the probability
$\theta$ of the null hypothesis being true. We can model this by assuming that
a success or failure is generated from a Bernoulli process that has
probability of success $\theta$:

$H_{0}\sim Bernoulli(\theta)$ (5)

Because our prior belief that the null is true will come with some uncertainty
(it is not merely a point value), we can model this prior belief through a
Beta distribution. For example, a prior $Beta(60,6)$ on $\theta$ expresses the
assumption that the prior probability of the null being true is between 0.83
and 0.97 with probability 95% (approximately), with mean probability
approximately 0.90. The lower and upper bounds of the 95% credible interval
can be computed using the inverse cumulative distribution function of the Beta
distribution. We simply solve the integrals for the lower and upper bounds:

$\int_{-\infty}^{lower}f(x)dx=\int_{-\infty}^{lower}\frac{\Gamma(a+b)}{\Gamma(a)\Gamma(b)}x^{a-1}(1-x)^{b-1}\,dx=0.025$
(6)

and

$\int_{-\infty}^{upper}f(x)dx=\int_{-\infty}^{upper}\frac{\Gamma(a+b)}{\Gamma(a)\Gamma(b)}x^{a-1}(1-x)^{b-1}\,dx=0.975$
(7)

The mean can be computed from the fact that a random variable $X$ that is
generated from a Beta distribution with parameters $a$ and $b$ has mean:

$E[X]=\frac{a}{a+b}$ (8)

Given the $a$ and $b$ parameters, the mean is:

$E[X]=\frac{a}{a+b}=\frac{60}{60+6}=0.9$ (9)

Figure 1b visualizes this prior probability of the null hypothesis being true.

Given this prior probability density function Beta(60,6) for $\theta$, we are
now in a position to investigate how the posterior probability of the null
being true changes under different assumptions. We can use Monte Carlo
sampling to compute the posterior probability of $H_{0}$ being true given
significant results under repeated sampling:

  1. 1.

Fix $\alpha$ (Type I error) and $\beta$ (Type II error).

  2. 2.

Do 100,000 times:

    1. (a)

Sample one value $\theta$ from the Beta(60,6) distribution.

    2. (b)

Compute posterior probability of $\theta$ given $\alpha,\beta$, and the
sampled value of $\theta$ from the prior distribution Beta(60,6):

$\alpha\times\frac{\theta}{(\alpha\times\theta+(1-\beta)\times(1-\theta))}$
(10)

    3. (c)

Store this posterior probability of the null hypothesis being true.

  3. 3.

Plot the distribution of the stored probabilities, or display summary
statistics such as the mean and the 95% credible interval.

There are two interesting cases. The first is when statistical power is low
(mean: 10%); we will show that in this case, it simply doesn’t matter much if
we get a significant result. The posterior probability of the null being true
will not change substantially; this is regardless of whether Type I error is
0.05 or some lower value like 0.005, as recommended by Benjamin . (2018). Note
that here, when we say that the posterior probability of the null doesn’t
change substantially, we are not talking about the mean of the posterior
probability; rather, we are more interested in the 95% credible interval,
which represents the uncertainty of the posterior probability. Thus, even if
the mean of the posterior distribution is numerically lower than the mean of
the prior distribution, if the uncertainty associated with the posterior
distribution is high, the prior-to-posterior change in mean probability is not
really meaningful.

The other case is where statistical power is high (mean: 90%); here, the
posterior probability of the null being true will change considerably once we
have a significant result under repeated sampling, especially if we follow the
recommendation of Benjamin . (2018) to lower Type I error to 0.005.

Anticipating our main conclusion, when the prior probability of the null being
true is low, the only situation where a statistically significant effect under
repeated sampling can shift our belief substantially against the null
hypothesis being true is when statistical power is high. When power is low, it
simply doesn’t matter whether you lower the Type I error to 0.005, as
suggested by Benjamin . (2018) and others. Null hypothesis significance
testing only makes sense if power is high; in all other situations, the
researcher is wasting their time computing p-values. When power is low, the
intuition, that a significant result will lead to a low posterior probability
of the null hypothesis being true, is an illusion.

Figure 2: Probabilities for the Null-hypothesis, $P(H_{0})=\theta$. Prior
probability (blue) and posterior probabilities given a significant effect
(grey) at a Type I error threshold $\alpha$ of $0.05$ and $0.01$. This is
shown for a situation of low statistical Power of $10$% (Type II error rate
$\beta=90$%) and with a high prior probability for the null hypothesis
($\theta\sim Beta(60,6)$; blue).

## Case 1: Investigating the posterior probability of the null hypothesis
being true when power is low (Type II error 0.90) and the prior probability of
the null hypothesis is high (Mean Prob(H0)=.90)

We next look at the posterior probability for different situations: As a first
case, we investigate the posterior probability of the null hypothesis being
true when the prior probability of the null hypothesis is high (Mean
Prob(H0)=.90) and when power is low (Type II error 0.90). We investigate
several scenarios by using different Type I errors ($\alpha=0.05,0.01$ and
$0.005$).

### Scenario 1: Low power (0.10), Type I error 0.05

Let Type I error be $\alpha=0.05$ and Type II error be $\beta=0.90$. So, we
have power at $1-\beta=0.10$. Such low power is by no means an uncommon
situation in areas like cognitive psychology, psycholinguistics, and
linguistics; examples from psycholinguistics are discussed in Jäger . (2017);
Vasishth . (2018); Nicenboim . (2018).

Figure 2 shows the prior and posterior distributions. The prior distribution
is plotted in blue. Figure 2 shows that getting a significant result hardly
shifts our belief regarding the null. This should be very surprising to
researchers who believe that a significant result shifts their belief about
the null hypothesis being true to a low value. Next, consider what happens
when we reduce Type I error to 0.01, which is lower than the traditional 0.05.

### Scenario 2: Low power (0.10), Type I error 0.01

Many researchers (Benjamin ., 2018) have suggested that lowering Type I error
will resolve many of the problems with NHST. Let’s start by investigating what
changes when we decrease Type I error to $0.01$ (researchers like Benjamin .,
2018 have proposed 0.005 as a threshold for Type I error; we turn to this
proposal below). Type II error is held constant at 0.90.

Figure 2 shows that lowering Type I error does shift our posterior probability
of the null being true a bit more but not enough to have any substantial
effect on our beliefs. It seems unreasonable to discard a null hypothesis if
the posterior probability of it being true lies between 30 and 70%.

### Scenario 3: Low power, Type I error 0.05, incorporating uncertainty about
Type II error

So far, we have been assuming a point value as representing power. However,
power is really a function that depends (inter alia) on the magnitude of the
true (unknown) effect. Power therefore also has some uncertainty associated
with it, because we do not know the magnitude of the true effect, and we do
not know the true standard deviation. We can introduce uncertainty about power
(or equivalently, uncertainty about Type II error) into the picture by setting
our prior on $\beta\sim Beta(10,4)$, so that the Type II error is around 70%.
Different levels of power (1-Type II error) are visualized in Figure 3, and
the low power situation of 30% is shown in the bottom row of the figure.

Figure 3: Visualization of the probability distribution associated with Type
II error $\beta$ corresponding to low power ($\beta\sim Beta(10,4)$, mean
power $E[1-\theta]=30$%), medium power ($\beta\sim Beta(8,8)$, mean power
$E[1-\theta]=50$%), and high power ($\beta\sim Beta(2,20)$, mean power
$E[1-\theta]=90$%) . Recall that Power is 1-Type II error. Figure 4:
Probabilities for the null hypothesis, $P(H_{0})=\theta$, considering
uncertainty about power. Prior probability (blue) and posterior probabilities
given a significant effect (grey) at a Type I error $\alpha$ of $0.05$ (upper
panels), $0.01$ (middle panels), and $0.005$ (lower panels). This is shown for
situations of low statistical Power, $\beta\sim Beta(10,4)$ (mean Type II
error rate of about $\beta=70$%, mean Power of about $30$%), medium
statistical Power, $\beta\sim Beta(8,8)$ (mean Type II error rate of
$\beta=50$%, mean Power of $50$%), and high statistical Power, $\beta\sim
Beta(2,20)$ (mean Type II error rate of about $\beta=10$%, mean Power of about
$90$%), and for situations with a high (left panels), medium (middle panels),
and low (right panels) prior probability for the null hypothesis.

Incorporating the uncertainty about Type II error (equivalently, power)
increases the uncertainty about the posterior probability of the null quite a
bit. Compare Figure 2 ($\alpha=0.05$) and Figure 4a (low power). Figure 4a
shows that the posterior of the null being true now lies between 40 and 90%
(as opposed to 70 and 90% in Figure 2).

### Scenario 4: Type I error 0.01, incorporating uncertainty in Type II error

Having incorporated uncertainty into Type II error, consider now what happens
if we lower Type I error to 0.01, from 0.05. Figure 4d shows (cf. low power)
that now the posterior distribution for the null hypothesis shifts to the left
quite a bit more, but with wide uncertainty (10-60%). Even with a low Type I
error of 0.01, we should be quite unhappy rejecting the null if the posterior
probability of the null being true is between the wide range of 10 and 60%.

### Scenario 5: Type I error 0.005, incorporating uncertainty in Type II error

Next, consider what happens if we lower Type I error to 0.005. This is the
suggestion from Benjamin . (2018). Perhaps surprisingly, Figure 4g shows that
now the posterior distribution for the null hypothesis does not shift much
compared to Scenario 4 (see Fig. 4d): the range is 6 to 45% (compare with the
range 10-60% in scenario 4). Thus, when power is low, there is simply no point
in engaging in null hypothesis significance testing. Simply lowering the
threshold of Type I error to 0.005 will not change much regarding our belief
in the null hypothesis.

## Case 2: Investigating the posterior probability of the null hypothesis
being true when power is high

As a second case, we investigate the posterior probability of the null
hypothesis being true when power is high. We consider the case where power is
around 90%. We will still assume a high prior probability for the null (Mean
Prob(H0) = .90). We will consider three scenarios: Type I error at 0.05, 0.01,
and 0.005.

### Scenario 6: High power (0.90), Type I error 0.05

First consider a situation where we have high power and Type I error is at the
conventional 0.05 value. The question here is: in high power situations, does
a significant effect shift our belief considerably away from the null, with
Type I error at the conventional value of 0.05? The prior on Type II error is
shown in Figure 3. The mean Type II error is 10%, implying a mean for the
power distribution to be 90%. Perhaps surprisingly, Figure 4a shows that even
under high power, our posterior probability of the null being true does not
shift dramatically: the probability lies between 20 and 60%.

### Simulation 7: High power (mean 0.90), Type I error 0.01

Next, we reduce Type I error to 0.01. Figure 4d shows that when power is high
and Type I error is set at 0.01, we get a big shift in posterior probability
of the null being true: the range is 5-25%.

### Simulation 8: High power (mean 0.90), Type I error 0.005

Next, in this high-power situation, we reduce Type I error to 0.005. Figure 4g
shows that when power is high and Type I error is set at 0.005, we get a
decisive shift in posterior probability of the null being true: the range is
now 2-13%.

## Cases 3 and 4: Prior probability for the null is medium or low

Finally, we consider cases where the prior probability for the null is medium
or low.

### Low prior probability for the null: Mean Prob(H0)=.10

One possible objection to the above analyses, however, is that the prior
probability for the null hypothesis could often be much smaller than an
average of 90%. Indeed, in some situations, the null hypothesis may be very
unlikely. We here simulate a situation where the prior probability for the
null is an average of 10% ($\theta\sim Beta(3,8)$). For this situation,
Figures 4c,f, and i show that the posterior probability for the null is always
decisively low. Even for a conventional Type I error of $\alpha=0.05$ in a
low-powered study, the posterior probability for the null ranges from 1 to
25%, which is quite low, and when turning to smaller $\alpha$ levels or higher
power, the effect is decisive.

However, of course this is not very informative, as we started out assuming
that the null hypothesis was unlikely to be correct in the first place. Thus,
we haven’t learned much; a statistical significance test would just confirm
what we already believed with high certainty before we carried out the test.

### Medium prior probability for the null: Mean Prob(H0)=.50

Now consider the case where the prior probability for the null being true lies
at an average of 50% (e.g., $\theta\sim Beta(15,15)$). Here, we don’t know
whether the null or the alternative hypothesis is true a priori, and both
outcomes seem similarly likely. In this situation, when we use a conventional
Type I error level of $\alpha=0.05$ in a low-powered study, a significant
effect will bring our posterior probability for the null only to a range of
6-40%, and will thus leave us with much uncertainty after obtaining a
significant effect.

However, either using a stricter Type I error level (e.g., $\alpha=0.005$) or
running a high-powered study each suffices to yield informative results: For a
high-powered study and $\alpha=0.05$, a significant result will (under our
assumptions) bring the posterior probability to 2-10% (Figure 4b), which is
quite informative. And for a Type I error level of $\alpha=0.005$ a
significant effect brings decisive evidence against the null for all the
levels of power that we investigated (Figure 4h), with a posterior probabilty
of 0.7-6% even for low-powered studies. This suggests that when the prior
probabilities of the null and the alternative hypotheses are each at 50%, then
either high power or a strict Type I error of $\alpha=0.005$ will yield
informative outcomes once a significant effect is observed.

## Discussion

In summary, we analyzed the posterior probability for the null given a
significant effect. We provide a shiny app
(https://danielschad.shinyapps.io/probnull/) that allows the user to compute
the posterior distribution for different choices of prior, Type I and Type II
error. For psychology and preclinical biomedical research, the prior odds of
H1 relative to H0 are estimated to be about 1:10 (Benjamin ., 2018),
reflecting a high prior probability of the null of 90%. For this common and
standard situation in psychology and other areas, when power is low, the
posterior probability of the null being true doesn’t change in any meaningful
way after seeing a significant result, even if we change Type I error to
0.005. What shifts our belief in a meaningful way is reducing Type I error to
say 0.005 (as suggested by Benjamin ., 2018 and others), as well as running a
high powered study. Only this combination of high power and small Type I error
rate yields informative results.

One might object here that we set the prior probability of the null hypothesis
being true at an unreasonably high value. This objection has some merit;
although typically the prior probability for the null may lie at 10%, there
may well be some situations where the null is unlikely to be true a priori. In
this situation, our results show that a significant effect does indicate a
very low posterior probability of the null. This is the case across a range of
Type I error levels ($\alpha$ of 0.05, 0.01, 0.005) and for different levels
of power (Figure 4c+f+i). Even for low power studies with $\alpha=0.05$ the
posterior probability is between 1 and 25%, which is quite low. So yes, if the
prior probability of the null being true is already low, even with relatively
low power and the standard Type I error level of 0.05, we are entitled to
changing our belief quite strongly against the null once we have a significant
effect. An obvious issue here is that if we already don’t believe in the null
before we do the statistical test, why bother to try to reject the null
hypothesis? Even if we were satisfied with rejecting a null hypothesis we
don’t believe in in the first place, running low power studies is always a bad
idea because of Type M and S error. As Gelman Carlin (2014) and many others
before them have pointed out, significant effects from low power studies will
have exaggerated estimates of effects and could have the wrong sign of the
effect. The probability of the null hypothesis being true is not the only
important issue in a statistical test; accurate estimation of the parameter of
interest and quantifying our uncertainty about that estimate are equally
important.

In summary, we investigated the intuitive belief held by some researchers that
finding a significant effect reduces the posterior probability of the null
hypothesis to a low value. We show that this intuition is not true in general.
The common situation in psychology and other areas is that the null hypothesis
is a priori quite likely to be true. In such a situation, contrary to
intuition, finding a significant effect leaves us with much posterior
uncertainty about the null hypothesis being true. Obtaining a reasonable
reduction in uncertainty is thus another reason to adopt the recent
recommendation by Benjamin . (2018) to change Type I error to $\alpha=0.005$.
Furthermore, conducting high power studies is an obvious but neglected remedy.
Otherwise, the results will be indecisive.

Our key result is that the posterior probability for the null given a
significant effect varies widely across settings involving different Type I
and Type II errors and different prior probabilities for the null. The
intuition that frequentist p-values may provide a shortcut to this information
is in general misleading.

## Acknowledgements

Thanks to Valentin Amrhein and Sander Greenland for comments. Funded by the
Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) – Project
number 317633480, SFB 1287, Project Q, with principal investigators Shravan
Vasishth and Ralf Engbert.

## Author contribution

SV had the idea for the paper. SV and DJS performed analyses. SV and DJS
generated the shiny app. SV and DJS wrote the paper.

## Availability of simulations and computer code

All the computer code used for the simulations reported in the present
manuscript, and the code for generating all Figures, will be freely available
online at https://osf.io/9g5bp/. Moreover, we make a shiny app available at
https://danielschad.shinyapps.io/probnull/ that allows computing the posterior
probability for the null given a significant effect for many different
settings of Type I and II error and for different prior probabilities for the
null.

## References

  * Benjamin . (2018) benjamin2018redefineBenjamin, DJ., Berger, JO., Johannesson, M., Nosek, BA., Wagenmakers, EJ., Berk, R.others  2018\.  Redefine statistical significance Redefine statistical significance.  Nature Human Behaviour216.
  * Betancourt (2018) betancourt2018calibratingBetancourt, M.  2018\.  Calibrating Model-Based Inferences and Decisions Calibrating model-based inferences and decisions.  arXiv preprint arXiv:1803.08393.
  * Doherty . (2002) doherty2002fluorideDoherty, UB., Benson, PE. Higham, SM.  2002\.  Fluoride-releasing elastomeric ligatures assessed with the in situ caries model Fluoride-releasing elastomeric ligatures assessed with the in situ caries model.  The European Journal of Orthodontics244371–378.
  * Gelman Carlin (2014) gelmancarlinGelman, A. Carlin, J.  2014\.  Beyond Power Calculations: Assessing Type S (Sign) and Type M (Magnitude) Errors Beyond power calculations: Assessing Type S (sign) and Type M (magnitude) errors.  Perspectives on Psychological Science96641–651.
  * Harris Taylor (2003) harristaylorHarris, M. Taylor, G.  2003\.  Medical statistics made easy Medical statistics made easy.  Scion Banbury, England.
  * Jäger . (2017) JaegerEngelmannVasishth2017Jäger, LA., Engelmann, F. Vasishth, S.  2017\.  Similarity-based interference in sentence comprehension: Literature review and Bayesian meta-analysis Similarity-based interference in sentence comprehension: Literature review and Bayesian meta-analysis.  Journal of Memory and Language94316-339.  https://doi.org/10.1016/j.jml.2017.01.004
  * Kolmogorov (2018) kolmogorov2018foundationsKolmogorov, AN.  2018\.  Foundations of the Theory of Probability: Second English Edition Foundations of the theory of probability: Second English Edition.  Courier Dover Publications.
  * McElreath (2016) mcelreath2016statisticalMcElreath, R.  2016\.  Statistical rethinking: A Bayesian course with examples in R and Stan Statistical rethinking: A Bayesian course with examples in R and Stan ( 122).  CRC Press.
  * Nicenboim . (2018) NicenboimRoettgeretalNicenboim, B., Roettger, TB. Vasishth, S.  2018\.  Using meta-analysis for evidence synthesis: The case of incomplete neutralization in German Using meta-analysis for evidence synthesis: The case of incomplete neutralization in German.  Journal of Phonetics7039-55.  https://osf.io/g5ndw/ https://doi.org/10.1016/j.wocn.2018.06.001
  * Oakley O’Hagan (2010) OakleyOHaganOakley, JE. O’Hagan, A.  2010\.  SHELF: The Sheffield Elicitation Framework (version 2.0) SHELF: The Sheffield Elicitation Framework (version 2.0) [].  University of Sheffield, UK.  http://tonyohagan.co.uk/shelf
  * O’Hagan . (2006) ohagan2006uncertainO’Hagan, A., Buck, CE., Daneshkhah, A., Eiser, JR., Garthwaite, PH., Jenkinson, DJ.Rakow, T.  2006\.  Uncertain judgements: Eliciting experts’ probabilities Uncertain judgements: Eliciting experts’ probabilities.  John Wiley & Sons.
  * Tam . (2018) tam2018doctorsTam, CWM., Khan, AH., Knight, A., Rhee, J., Price, K., McLean, K. .  2018\.  How doctors conceptualise’P’values: A mixed methods study How doctors conceptualise’p’values: A mixed methods study.  Australian Journal of General Practice4710705.
  * Vasishth . (2018) VasishthMertzenJaegerGelman2018Vasishth, S., Mertzen, D., Jäger, LA. Gelman, A.  2018\.  The statistical significance filter leads to overoptimistic expectations of replicability The statistical significance filter leads to overoptimistic expectations of replicability.  Journal of Memory and Language103151-175.  https://osf.io/eyphj/ https://doi.org/10.1016/j.jml.2018.07.004


At each scan cycle $\mathsf{Ctrl}$ checks if the value of temperature as
carried by channel $\mathit{ch\\_temp}$ is too high, namely above
$\mathsf{threshold}$, which is a constant $\leq\mathsf{max}$ and set to 99.8
in our experiments. If this is the case, then $\mathsf{Ctrl}$ activates the
coolant for $5$ consecutive time instants, otherwise the coolant remains off.
In both cases, before re-starting its scan cycle, $\mathsf{Ctrl}$ waits for
instructions/requests to change the speed: if it receives instructions through
channel $\mathit{ch\\_speed}$ from the IDS to slow down the engine then it
commands so through actuator $\mathit{speed}$. Otherwise, if the IDS of the
other engine requests through channel $\mathit{ch\\_in}$ to work at
$\mathsf{full}$ of $\mathsf{half}$ power, then $\mathsf{Ctrl}$ sets
$\mathit{speed}$ accordingly.

The process $\mathsf{IDS}$ checks whether the cooling system is active when
the temperature is above $\mathsf{max}$. If this safety condition is violated
then: [(i)]

it raises the presence of an anomaly, via channel $\mathit{ch\\_warning}$;

it commands $\mathsf{Ctrl}$ to slow down the engine, via
$\mathit{ch\\_speed}$;

it requests to the other engine to run at full power, via channel
$\mathit{\mathit{ch}\\_{\mathit{out}}}$. Otherwise, $\mathsf{IDS}$ asks (both
local and external) controllers to set their engines at $\mathsf{half}$ power.

An engine system can be built by combining two engines, the “left” and the
“right” one, interacting with a supervisory component that checks their
correct functioning. The composite system can be defined as

$\mathsf{Eng\\_Sys}\;=\;\mathsf{Eng}\\_{\mathsf{L}}~{}|~{}\mathsf{Eng}\\_{\mathsf{R}}~{}|~{}\mathsf{SV}$

where processes $\mathsf{Eng}\\_{\mathsf{L}}$ and
$\mathsf{Eng}\\_{\mathsf{R}}$ are obtained from $\mathsf{Eng}$ by renaming the
variables as follows: [(i)]

both $\mathit{ch\\_in}$ in $\mathsf{Eng\\_L}$ and $\mathit{ch\\_out}$ in
$\mathsf{Eng\\_R}$ are renamed as $\mathit{ch\\_speed\\_R\\_to\\_L}$;

both $\mathit{ch\\_in}$ in $\mathsf{Eng\\_R}$ and $\mathit{ch\\_out}$ in
$\mathsf{Eng\\_L}$ are renamed as $\mathit{ch\\_speed\\_L\\_to\\_R}$;

all remaining variables $\mathit{x}$ used in $\mathit{Eng}$ are renamed as
$\mathit{x\\_L}$ in $\mathsf{Eng\\_L}$ and as $\mathit{x\\_R}$ in
$\mathsf{Eng\\_R}$. Then, we introduce a new channel $\mathit{ch\\_alarm}$
which is used by $\mathsf{SV}$ to forward to the external environment the
information obtained from the IDSs through $\mathit{ch\\_warning\\_L}$ and
$\mathit{ch\\_warning\\_R}$. In detail, $\mathit{ch\\_alarm}$ carries a value
in the set $\\{\mathsf{none},\mathsf{left},\mathsf{right},\mathsf{both}\\}$
depending on the anomaly warnings received from the left and right IDS:

$\begin{array}[]{lll}\mathsf{SV}&\stackrel{{\scriptstyle\mathit{def}}}{{=}}&\mathrm{if}\;[\mathit{ch\\_warning\\_L}=\mathsf{hot}\wedge\mathit{ch\\_warning\\_R}=\mathsf{hot}]\;(\mathsf{both}\rightarrow\mathit{ch\\_alarm}).\mathsf{SV}\\\
&&\mathrm{else}\;\mathrm{if}\;[\mathit{ch\\_warning\\_L}=\mathsf{hot}]\;(\mathsf{left}\rightarrow\mathit{ch\\_alarm}).\mathsf{SV}\\\
&&\;\;\;\;\;\;\;\mathrm{else}\;\mathrm{if}\;[\mathit{ch\\_warning\\_R}=\mathsf{hot}]\;(\mathsf{right}\rightarrow\mathit{ch\\_alarm}).\mathsf{SV}\\\
&&\;\;\;\;\;\;\;\;\;\;\;\;\;\;\mathrm{else}\;(\mathsf{none}\rightarrow\mathit{ch\\_alarm}).\mathsf{SV}\end{array}$

Following [LMMT21], cyber-physical attacks can be modelled as processes that
run in parallel with systems under attack. Intuitively, they represent
malicious code that has been injected in the logic of the system. Three
examples of attack to $\mathsf{Eng\\_Sys}$ are the following, where
$\mathsf{X}\in\\{\mathsf{L},\mathsf{R}\\}$:

$\begin{array}[]{lll}\mathsf{Att\\_Act\\_X}&\stackrel{{\scriptstyle\mathit{def}}}{{=}}&\mathrm{if}\;[\mathit{temp\\_X}<\mathsf{max}-\mathsf{AC}]\;(\mathsf{off}\rightarrow\mathit{cool\\_X}).\mathsf{Att\\_Act\\_X}\;\mathrm{else}\;\surd.\mathsf{Att\\_Act\\_X}\\\
\mathsf{Att\\_Sen\\_X}&\stackrel{{\scriptstyle\mathit{def}}}{{=}}&(-\mathsf{TF}\rightarrow\mathit{temp\\_fake\\_X}).\mathsf{Att\\_Sen\\_X}\\\
\mathsf{Att\\_Saw\\_X}&\stackrel{{\scriptstyle\mathit{def}}}{{=}}&((\mathit{left}\rightarrow\mathit{AW\\_l}),(\mathit{right}\rightarrow\mathit{AW\\_r})).\mathsf{Att\\_Saw\\_X^{\prime}}\\\
\mathsf{Att\\_Saw\\_X^{\prime}}&\stackrel{{\scriptstyle\mathit{def}}}{{=}}&\mathrm{if}\;[\mathit{cs}\in[\mathit{AW_{l}},\mathit{AW}_{r}]\;(-\mathsf{TF}\rightarrow\mathit{temp\\_fake\\_X}).\mathsf{Att\\_Saw\\_X^{\prime}}\\\
&&\mathrm{else}\;(0\rightarrow\mathit{temp\\_fake\\_X}).\mathsf{Att\\_Saw\\_X^{\prime}}\end{array}$

Process $\mathsf{Att\\_Act\\_X}$ models an integrity attack on actuator
$\mathit{cool\\_X}$. The five-instants cooling cycle started by
$\mathsf{Ctrl\\_X}$ is maliciously interrupted as soon as $\mathit{temp\\_X}$
goes below $\mathsf{max}-\mathsf{AC}$, with $\mathsf{AC}$ a constant with
value in $[1.0,3.0]$, in order to let the temperature of the engine rise
quickly above $\mathsf{max}$, accumulating stress in the system, and requiring
continuous activation of the cooling system. We recall that the quantification
of stress accumulated by engine $\mathit{Eng\\_X}$ is recorded in variable
$\mathit{stress\\_X}$, which is incremented by $\mathcal{E}$ when $|\\{k\mid
1\leq k\leq 6\wedge p_{k}\geq\mathsf{max}\\}|>3$. In [LMMT21] this attack is
classified as _stealth_ , since the condition
$\mathit{temp\\_X}>\mathsf{max}\wedge\mathit{cool\\_X}=\mathsf{off}$ monitored
by the IDS never holds. Notice that $\mathsf{AC}$ is a parameter of the
attack.

Then, $\mathsf{Att\\_Sen\\_X}$ is an integrity attack to sensor
$\mathit{temp\\_X}$. The attack adds a negative offset $-\mathsf{TF}$, with
$\mathsf{TF}$ a positive constant, to the temperature value carried by
$\mathit{ch\\_temp\\_X}$, namely $ch\\_temp\\_X$ becomes
$\mathit{temp\\_X}-\mathsf{TF}$. This attack aims to prevent some activations
of the cooling system by $\mathsf{Ctrl\\_X}$. Since the IDS raises a warning
on $\mathit{ch\\_warning\\_X}$ each time we have
$\mathsf{max}<\mathit{temp\\_X}\leq\mathsf{max}+\mathsf{TF}$ and
$\mathit{cool\\_X}=\mathsf{off}$, this attack is not stealth. In this case,
$\mathsf{TF}$ is a parameter of the attack.

Finally, $\mathsf{Att\\_Saw\\_X}$ performs the same attack of
$\mathsf{Att\\_Sen\\_X}$ but only in a precise attack window. We assume that
$\mathit{cs}$ is a variable that simply counts the number of computation steps
(the initial value is $0$ and $\mathit{cs}(\tau+1)=\mathit{cs}(\tau)+1$ is
added to Equation 15). Then, $[\mathit{AW_{l}},\mathit{AW_{r}}]$ is an
interval, where $\mathit{AW_{l}}=0=\mathit{AW_{r}}$ in the initial data state
and we assume that the variables $\mathit{left}$ and $\mathit{right}$ take
random non-negative integer values satisfying the property
$0\leq\mathit{right}-\mathit{left}\leq\mathsf{AWML}$, where $\mathsf{AWML}$ is
a parameter of the attack representing the maximal length of the attack
window.

###  7.2. Simulation of the engine system

We have applied our simulation strategy to the engine system. Figure 14
reports the results of six simulations of the single engine
$\mathsf{Eng\\_L}$, where each simulation is obtained by simulating $100$ runs
consisting in $10000$ steps, with the following initial data state
$\mathbf{d}_{0}$: [(i)]

$\mathit{temp\\_L}=\mathit{p}_{1}\\_L=\dots=\mathit{p}_{6}\\_L=95$;

$\mathit{cool\\_L}=\mathsf{off}$;

$\mathit{speed\\_L}=\mathsf{half}$;

$\mathit{stress\\_L}=0$;

$\mathit{ch\\_speed\\_L}$ = $\mathit{ch\\_speed\\_L\\_to\\_R}$=
$\mathit{ch\\_speed\\_R\\_to\\_L}$= $\mathsf{half}$;

$\mathit{temp\\_fake\\_L}=0$; and

$\mathit{ch\\_warning\\_L}=\mathsf{ok}$. We considered three different
scenarios: [(i)]

no attack, modelled by processes $\mathsf{Eng\\_L}$,

attack on actuator $\mathit{cool\\_L}$, modelled by
$\mathsf{Eng\\_L}~{}|~{}\mathsf{Att\\_Act\\_L}$, with constant $\mathsf{AC}$
set to $1.8$, and

attack on sensor $\mathit{temp\\_L}$, modelled by
$\mathsf{Eng\\_L}~{}|~{}\mathsf{Att\\_Sen\\_L}$, with constant $\mathsf{TF}$
set to $0.4$. The six pictures report the value of average temperature
detected by sensor $\mathit{temp\\_L}$ and the average stress level carried by
variable $\mathit{stress\\_L}$ in these three scenarios.

Clearly, by comparing the plots in Figure 14 it turns out that both attacks
are able to cause an average size-up of the temperature of the engine and to
take the engine to the maximal level of stress after 10000 steps.

Figure 14. Average temperature (sensor $\mathit{temp\\_L}$) and stress
(variable $\mathit{stress\\_L}$) for left engine running in three scenarios:
no attack, attack on actuator $\mathit{cool\\_L}$, attack on sensor
$\mathit{temp\\_L}$.
Simulation settings: $k=10000$ steps, $N=100$ runs.
Initial sensor/actuator settings: $\mathit{temp\\_L}=95$,
$\mathit{cool\\_L}=\mathsf{off}$, $\mathit{speed\\_L}=\mathsf{half}$.
Values assigned to constants: $\mathsf{AC}=1.8$, $\mathsf{TF}=0.4$.

###  7.3. Analysis of the engine system

Our aim is to use evolution metrics in order to study the impact of the
attacks. This requires to compare the evolution sequences of the genuine
system and those of the system under attack. The two approaches to the
quantification of impact of attacks outlined above, consisting in evaluating
only damages caused by attacks or both damages and alert signals, will be
realised by employing different penalty functions.

We start with quantifying the impact of cyber-physical attacks by focusing on
the runtime values of _false negatives_ and _false positives_. Intuitively, if
we focus on a single engine $\mathsf{Eng\\_X}$, false negatives and false
positives represent the average _effectiveness_ , and the average _precision_
, respectively, of the IDS modelled by $\mathsf{IDS\\_X}$ to signal through
channel $\mathit{ch\\_warning\\_X}$ that the engine is under stress. In our
setting, we can add two variables $\mathit{fn\\_X}$ and $\mathit{fp\\_X}$ that
quantify false negatives and false positives depending on
$\mathit{stress\\_X}$ and $\mathit{ch\\_warning\\_X}$. Both these variables
are updated by $\mathcal{E}$. In detail, $\mathcal{E}$ acts on
$\mathit{fn\\_X}$ and $\mathit{fp\\_X}$ as follows:

$\begin{array}[]{lll}\mathit{fn\\_X}(\tau+1)&=&\frac{\tau*\mathit{fn\\_X}(\tau)+\max(0,\mathit{stress\\_X(\tau)}-\mathit{ch\\_warning\\_X(\tau)})}{\tau+1}\\\\[4.30554pt]
\\\
\mathit{fp\\_X}(\tau+1)&=&\frac{\tau*\mathit{fp\\_X}(\tau)+\max(0,\mathit{ch\\_warning\\_X(\tau)}-\mathit{stress\\_X}(\tau))}{\tau+1}\end{array}$

where $\mathit{ch\\_warning\\_X}\in\\{\mathsf{hot},\mathsf{ok}\\}$ is viewed
as a real by setting $\mathsf{hot}=1$ and $\mathsf{ok}=0$. In this way, as
reported in [LMMT21], $\mathit{fn\\_X}$ and $\mathit{fp\\_X}$ carry the
average value, in probability and time, of the difference between the value of
the stress of the system and of the warning raised by the IDS.

False negatives and false positives for the same simulations described in
Figure 14 are reported in Figure 15.

Figure 15. Average false negatives (variable $\mathit{fn\\_L}$) and false
positives (variable $\mathit{fp\\_L}$) for the simulations of Figure 14.

Clearly, if we take the value of variable $\mathit{fn\\_X}$ (resp.
$\mathit{fp\\_X}$) as the value returned by our penalty function, we can
quantify how worse $\mathit{Eng\\_X}$ under attack behaves with respect to the
genuine version of $\mathit{Eng\\_X}$, where the evaluation of these systems
is done by taking into account their false negatives (resp. false positives).
Notice that other approaches are possible. For instance, by defining a penalty
function returning a convex combination of the values of $\mathit{fn\\_X}$ and
$\mathit{fp\\_X}$, systems are evaluated by suitable weighting their false
negatives and false positives.

In Figure 16 we report the distances between the genuine system
$\mathsf{Eng\\_L}$ and the compromised systems
$\mathsf{Eng\\_L}~{}|~{}\mathsf{Att\\_Act\\_L}$ and
$\mathsf{Eng\\_L}~{}|~{}\mathsf{Att\\_Sen\\_L}$ in terms of false negatives
and false positives. The settings for these simulations are the same of those
used for the simulations described in Figure 14.

(a) $\mathit{fn\\_L}$ distance, attack on $\mathsf{cool\\_L}$

(b) $\mathit{fn\\_L}$ distance, attack on $\mathsf{temp\\_L}$

(c) $\mathit{fp\\_L}$ distance, attack on $\mathsf{cool\\_L}$

(d) $\mathit{fp\\_L}$ distance, attack on $\mathsf{temp\\_L}$

Figure 16. False negative and false positive distance between genuine
$\mathsf{Eng\\_L}$ and $\mathsf{Eng\\_L}$ under actuator or sensor attack, for
the simulations of Figure 14.

If we move to the whole system, the false negatives and false positives are
carried by variables $\mathit{fn}$ and $\mathit{fp}$, that are updated by
$\mathcal{E}$ as follows:

$\begin{array}[]{lll}\mathit{fn}(\tau+1)&=&\frac{\tau*\mathit{fn}(\tau)+\min(1,(0.7*\max(0,\mathit{stress\\_L(\tau)}-\mathit{l})+0.7*\max(0,\mathit{stress\\_R(\tau)}-\mathit{r})))}{\tau+1}\\\\[2.15277pt]
\mathit{fp}(\tau+1)&=&\frac{\tau*\mathit{fp}(\tau)+\min(1,(0.7*\max(0,\mathit{l}-\mathit{stress\\_L}(\tau))+0.7*\max(0,\mathit{r}-\mathit{stress\\_R}(\tau))))}{\tau+1}\end{array}$

where $l=\begin{cases}1&\text{if
}\mathit{ch\\_alarm}(\tau)\in\\{\mathsf{both},\mathsf{left}\\}\\\ 0&\text{if
}\mathit{ch\\_alarm}(\tau)\in\\{\mathsf{none},\mathsf{right}\\}\end{cases}$
and $r=\begin{cases}1&\text{if
}\mathit{ch\\_alarm}(\tau)\in\\{\mathsf{both},\mathsf{right}\\}\\\ 0&\text{if
}\mathit{ch\\_alarm}(\tau)\in\\{\mathsf{none},\mathsf{left}\\}\end{cases}$.

Here, false negatives and false positives represent the average effectiveness,
and the average precision, respectively, of the supervisors modelled by
$\mathsf{SV}$ to signal through channel $\mathit{ch\\_alarm}$ that the system
is under stress. We remark that the values 0.7 are the values chosen in
[LMMT21] to quantify the _security clearance_ of the two engines, where, in
general, the clearance of a CPS component quantifies the importance of that
component from a security point of view. In this case, the two engines have
the same criticality and, consequently, the same security clearance.

Figure 17. False negative and false positive distance between genuine
$\mathsf{Eng\\_Sys}$ and $\mathsf{Eng\\_Sys}$ under attack on left sensor and
right actuator.

In Figure 17 we report the distance, in terms of false negatives and false
positives, between the system $\mathsf{Eng\\_Sys}$ and the compromised system
$\mathsf{Eng\\_Sys}~{}|~{}\mathsf{Att\\_Act\\_L}~{}|~{}\mathsf{Att\\_Sen\\_R}$
that is subject to both an attack on the actuator $\mathit{cool\\_L}$ of the
left engine and an attack on the sensor $\mathit{temp\\_R}$ of the right
engine. Again, the settings for these simulations are the same of those used
for simulations described in Figure 14.

Let us focus now on the third attack, namely the one implemented by process
$\mathsf{Att\\_Saw\\_X}$. Intuitively, we expect a correlation between the
length of the attack window and the impact of the attack. In order to study
such a relation, we may proceed as follows: [(i)]

we define a penalty function $\rho$ so that its value depends on the length of
the attack window $[\mathit{AW_{l}},\mathit{AW_{r}}]$. Actually, we may define
$\rho$ to return
$\nicefrac{{(\mathit{AW_{r}}-\mathit{AW_{l}})}}{{\mathsf{AWML}}}$. In
particular, if we consider the genuine system then $\rho$ returns $0$, since
$\mathit{AW_{l}}=0=\mathit{AW_{r}}$.

we consider a penalty function $\rho^{\prime}$ expressing the impact of the
attack.

we estimate the $(\rho,\rho^{\prime},[0,0],0,\eta_{1},\eta_{2})$-robustness of
the genuine system. More in detail, we may sample $M$ variations of the system
at $\rho$-distance from the genuine one bounded by $\eta_{1}$. This means
sampling $M$ versions of $\mathsf{Eng\\_X~{}|~{}Att\\_Saw\\_X}$ characterised
by attack windows of length bounded by $\eta_{1}\cdot\mathsf{AWML}$. Then, by
applying our simulations, we can estimate how worse these compromised systems
behave with respect to the genuine system $\mathsf{Eng\\_X}$, according to the
impact expressed by $\rho^{\prime}$.

In Figure 18 we report the results of such an estimation for $\rho^{\prime}$
coinciding with $\mathit{fn\\_L}$, $[\mathit{AW_{l}},\mathit{AW_{r}}]=[0,aw]$
an attack window of length $aw$ starting at first instant, $\mathsf{AWML}$ set
to 1000 and $\eta_{1}=0.1$, which means that $aw$ is chosen probabilistically
according to a uniform distribution on interval $[0,100]$.

Here we consider three different offset values assigned to $\mathsf{TF}$: 0.4,
0.6, and 0.8. In all cases, we estimate to have robustness, for values for
$\eta_{2}$ that are $0.012$, $0.05$, and $0.11$, respectively.

(a) $\mathit{TF}=0.4$

(b) $\mathit{TF}=0.6$

(c) $\mathit{TF}=0.8$

Figure 18. Estimation of false negative distance between a genuine engine and
an engine subject to tampering with sensor $\mathit{temp\\_L}$. The attacker
adds a negative offset $-0.4$ (Fig. 18(a)), or $-0.6$ (Fig. 18(b)), or $-0.8$
(Fig. 18(c)) to sensor, on the attack window $[0,aw]$, with $aw$ an integer
uniformly distributed in $[0,100]$. Simulation settings for each offset:
$M=20$ different sampled values for each $aw$, $N=100$ runs by the genuine
engine and $N*l=1000$ runs by the compromised system for each $aw$, $k=10000$
steps per run.

Now, let us change the approach for attack quantification and assume that only
the inflicted overstress is considered, without taking into account the raised
alarm signals. In particular, this overstress is quantified at a given
computation step $n$ as the ratio between the number of computation steps
where the stress condition $|\\{k\mid\mathit{p_{k}}\geq\mathsf{max}\\}|>3$
holds and $n$. Therefore, the penalty function $\rho^{\prime}$ is defined now
as the ratio between the values of two variables, one counting the number of
instants with the stress condition and one counting the total number of steps.
Robustness can be estimated as in the earlier case. In Figure 19 we report the
robustness of the left engine $\mathsf{Eng\\_L}$. Here we use the same
simulation settings used in Figure 18 and the same offset values. The figure
shows that we have robustness for low values of $\eta_{2}$, which, in essence,
tells us that attacks in a given window are unable to generate overstress in
the long run.

(a) $\mathit{TF}=0.4$

(b) $\mathit{TF}=0.6$

(c) $\mathit{TF}=0.8$

Figure 19. Estimation of average stress distance between a genuine engine and
an engine subject to tampering with sensor $\mathit{temp\\_L}$. The simulation
settings are the same of Figure 18.

##  8\. Concluding Remarks

As a first step for future research we will provide a simple logic, defined in
the vein of _Signal Temporal Logic_ (STL) [MN04], that can be used to specify
requirements on the evolution sequences of a system. Our intuition is that we
can exploit the evolution metric, and the algorithm we have proposed, to
develop a quantitative model checking tool for this type of systems. Moreover,
we would like to enhance the modelling of time. Firstly we could relax the
timing constraints on the evolution metric by introducing a _time tolerance_
and defining a _stretched evolution metric_ as a Skorokhod-like metric
[Sko56], as those used for conformance testing [DMP17]. Then, we could provide
an extension of our techniques to the case in which also the program shows a
continuous time behaviour.

The use of metrics for the analysis of systems stems from [GJS90, DGJP04,
KN96] where, in a process algebraic setting, it is argued that metrics are
indeed more informative than behavioural equivalences when quantitative
information on the behaviour is taken into account. The Wasserstein lifting
has then found several successful applications: from the definition of
_behavioural metrics_ (e.g., [vB05, CLT20b, GLT16]), to privacy [CGPX14,
CCP18, CCP20] and machine learning (e.g., [ACB17, GAA+17, TBGS18]). Usually,
one can use behavioural metrics to quantify how well an implementation ($I$)
meets its specification ($S$). In [CHR12] the authors do so by setting a two
players game with weighted choices, and the cost of the game is interpreted as
the distance between $I$ and $S$. Hence the authors propose three distance
functions: _correctness_ , _coverage_ , and _robustness_. Correctness
expresses how often $I$ violates $S$, coverage is its dual, and robustness
measures how often $I$ can make an unexpected error with the resulting
behaviour still meeting $S$. A similar game-based approach is used in [CDDP19]
to define a _masking fault-tolerant_ distance. Briefly, a system is masking
fault-tolerant if faults do not induce any observable behaviour in the system.
Hence, the proposed distance measures how many faults are tolerated by $I$
while being masked by the states of the system. Notice that the notion of
robustness from [CHR12] and the masking fault-tolerant distance from [CDDP19]
are quite different from our notions of reliability and robustness. In fact,
we are not interested in counting how many times an error occurs, but in
checking whether the system is able to regain the desired behaviour after the
occurrence of an error.

One of the main objectives in the control theory research area is the
synthesis of controllers satisfying some desired property, such as safety,
stability, robustness, etc. Conversely, our purpose in this paper was to
provide some tools for the _analysis_ of the interaction of a _given_ program
with an environment. The idea is that these tools can be used to test a
synthesised controller against its deployment in various environments. While a
direct application of our framework to the synthesis of programs does not seem
feasible, it would be interesting to investigate whether a combination of it
with learning techniques can be used for the design and synthesis of robust
controllers. Our confidence in a potential application relies on some recently
proposed metric-based approaches to controllers synthesis [GBS+19, AP11], in
which, however, the environment is only modelled deterministically.

To conclude, we remark that our evolution sequences are not a rewriting of
Markov processes as _transformers of distributions_ [KA04, KVAK10]. Roughly,
in the latter approach, one can interpret state-to-state transition
probabilities as a single distribution over the state space, so that the
behaviour of the system is given by the sequence of the so obtained
distributions. While the two approaches may seem similar, there are some
substantial differences. Firstly, the state space in [KA04, KVAK10] is finite
and discrete, whereas here we are in the continuous setting. Secondly, the
transformers of distributions consider the behaviour of the system as a whole,
i.e., it is not possible to separate the logical component from the
environment.

## Acknowledgements

V. Castiglioni has been supported by the project ‘ _Open Problems in the
Equational Logic of Processes_ ’ (OPEL) of the Icelandic Research Fund (grant
No. 196050-051).

## References

  * [ACB17] Martín Arjovsky, Soumith Chintala, and Léon Bottou.  Wasserstein generative adversarial networks.  In Proceedings of ICML 2017, pages 214–223, 2017.
  * [ADB11] Alessandro Abate, Alessandro D’Innocenzo, and Maria Domenica Di Benedetto.  Approximate abstractions of stochastic hybrid systems.  IEEE Trans. Automat. Contr., 56(11):2688–2694, 2011.
  * [AGA+17] Daniele Antonioli, Hamid Reza Ghaeini, Sridhar Adepu, Martín Ochoa, and Nils Ole Tippenhauer.  Gamifying ICS security training and research: Design, implementation, and results of S3.  In Proceedings of CPS-SPC@CCS 2017, pages 93–102. ACM, 2017\.
  * [AKLP10] Alessandro Abate, Joost-Pieter Katoen, John Lygeros, and Maria Prandini.  Approximate model checking of stochastic hybrid systems.  Eur. J. Control, 16(6):624–641, 2010.
  * [AL92] Karl J. Astrom and Michael Lundh.  Lund control program combines theory with hands-on experience.  IEEE control systems, 3(12):22–30, 1992.
  * [ALGG+06] Ignacio Alvarado, Daniel Limon, Winston García-Gabín, Teodoro Alamo, and Eduardo F. Camacho.  An educational plant based on the quadruple-tank process.  IFAC Proceedings Volumes, 39(6):82–87, 2006.  7th IFAC Symposium on Advances in Control Education.
  * [AP11] Alessandro Abate and Maria Prandini.  Approximate abstractions of stochastic systems: A randomized method.  In Proceedings of CDC-ECC 2011, pages 4861–4866, 2011.
  * [BNL13] Marco Bernardo, Rocco De Nicola, and Michele Loreti.  A uniform framework for modeling nondeterministic, probabilistic, stochastic, or mixed processes and their behavioral equivalences.  Inf. Comput., 225:29–82, 2013.
  * [Bog07] Vladimir I. Bogachev.  Measure Theory.  Number v. 1 in Measure Theory. Springer-Verlag, Berlin/Heidelberg, 2007\.
  * [CCP18] Valentina Castiglioni, Konstantinos Chatzikokolakis, and Catuscia Palamidessi.  A logical characterization of differential privacy via behavioral metrics.  In Proceedings of FACS 2018, volume 11222 of Lecture Notes in Computer Science, pages 75–96, 2018.
  * [CCP20] Valentina Castiglioni, Konstantinos Chatzikokolakis, and Catuscia Palamidessi.  A logical characterization of differential privacy.  Sci. Comput. Program., 188:102388, 2020.
  * [CDDP19] Pablo F. Castro, Pedro R. D’Argenio, Ramiro Demasi, and Luciano Putruele.  Measuring masking fault-tolerance.  In Proceedings of TACAS 2019, Part II, volume 11428 of Lecture Notes in Computer Science, pages 375–392, 2019.
  * [CGPX14] Konstantinos Chatzikokolakis, Daniel Gebler, Catuscia Palamidessi, and Lili Xu.  Generalized bisimulation metrics.  In Proceedings of CONCUR 2014, pages 32–46, 2014.
  * [CH08] Federica Ciocchetta and Jane Hillston.  Bio-pepa: An extension of the process algebra PEPA for biochemical networks.  Electron. Notes Theor. Comput. Sci., 194(3):103–117, 2008.
  * [CHR12] Pavol Cerný, Thomas A. Henzinger, and Arjun Radhakrishna.  Simulation distances.  Theor. Comput. Sci., 413(1):21–35, 2012.
  * [CLe07] Christos G. Cassandras, John Lygeros, and (eds.).  Stochastic Hybrid Systems.  Number 24 in Control Engineering. CRC Press, Boca Raton, 1st edition, 2007.
  * [CLT20a] Valentina Castiglioni, Michele Loreti, and Simone Tini.  Measuring adaptability and reliability of large scale systems.  In Proceedings of ISoLA 2020, volume 12477 of Lecture Notes in Computer Science, pages 380–396, 2020.
  * [CLT20b] Valentina Castiglioni, Michele Loreti, and Simone Tini.  The metric linear-time branching-time spectrum on nondeterministic probabilistic processes.  Theor. Comput. Sci., 813:20–69, 2020.
  * [dAHM03] Luca de Alfaro, Thomas A. Henzinger, and Rupak Majumdar.  Discounting the Future in Systems Theory.  In Proceedings of ICALP’03, ICALP ’03, pages 1022–1037. Springer, 2003.
  * [DGJP04] Josee Desharnais, Vineet Gupta, Radha Jagadeesan, and Prakash Panangaden.  Metrics for labelled Markov processes.  Theor. Comput. Sci., 318(3):323–354, 2004.
  * [DMP17] Jyotirmoy V. Deshmukh, Rupak Majumdar, and Vinayak S. Prabhu.  Quantifying conformance using the skorokhod metric.  Formal Methods in System Design, 50(2-3):168–206, 2017.
  * [FR18] Olivier P. Faugeras and Ludeger Rüschendorf.  Risk excess measures induced by hemi-metrics.  Probability, Uncertainty and Quantitative Risk, 3:6, 2018.
  * [GAA+17] Ishaan Gulrajani, Faruk Ahmed, Martín Arjovsky, Vincent Dumoulin, and Aaron C. Courville.  Improved training of Wasserstein GANs.  In Proceedings of Advances in Neural Information Processing Systems, pages 5767–5777, 2017.
  * [GBS+19] Shromona Ghosh, Somil Bansal, Alberto L. Sangiovanni-Vincentelli, Sanjit A. Seshia, and Claire Tomlin.  A new simulation metric to determine safe environments and controllers for systems with unknown dynamics.  In Proceedings of HSCC 2019, pages 185–196, 2019.
  * [GGI+15] Dieter Gollmann, Pavel Gurikov, Alexander Isakov, Marina Krotofil, Jason Larsen, and Alexander Winnicki.  Cyber-Physical Systems Security: Experimental Analysis of a Vinyl Acetate Monomer Plant.  In Proceedings of CPSS 2015, pages 1–12. ACM, 2015.
  * [GJS90] Alessandro Giacalone, Chi-Chang Jou, and Scott A. Smolka.  Algebraic reasoning for probabilistic concurrent systems.  In Proceedings of IFIP Work, Conf. on Programming, Concepts and Methods, pages 443–458, 1990.
  * [GLT16] Daniel Gebler, Kim G. Larsen, and Simone Tini.  Compositional bisimulation metric reasoning with probabilistic process calculi.  Log. Methods Comput. Sci., 12(4), 2016.
  * [HHH+94] Jane Hillston, Holger Hermanns, Ulrich Herzog, Vassilis Mertsiotakis, and Michael Rettelbach.  Stochastic process algebras: integrating qualitative and quantitative modelling.  In Proceedings of the 7th IFIP WG6.1 International Conference on Formal Description Techniques 1994, volume 6 of IFIP Conference Proceedings, pages 449–451, 1994.
  * [HJS+14] Guillermo Heredia, A. E. Jimenez-Cano, I. Sánchez, Domingo Llorente, V. Vega, J. Braga, José Ángel Acosta, and Aníbal Ollero.  Control of a multirotor outdoor aerial manipulator.  In Proceedings of IROS 2014, pages 3417–3422. IEEE, 2014.
  * [HLS00] Jianghai Hu, John Lygeros, and Shankar Sastry.  Towars a theory of stochastic hybrid systems.  In Proceedings of HSCC 2000, volume 1790 of Lecture Notes in Computer Science, pages 160–173, 2000.
  * [Joh00] Karl H. Johansson.  The quadruple-tank process: a multivariable laboratory process with an adjustable zero.  IEEE Trans. Control. Syst. Technol., 8(3):456–465, 2000.
  * [KA04] YoungMin Kwon and Gul Agha.  Linear inequality LTL (iltl): A model checker for discrete time markov chains.  In Proceedings of ICFEM 2004, volume 3308 of Lecture Notes in Computer Science, pages 194–208, 2004.
  * [KN96] Marta Z. Kwiatkowska and Gethin Norman.  Probabilistic metric semantics for a simple language with recursion.  In Proceedings of MFCS’96, volume 1113 of Lecture Notes in Computer Science, pages 419–430, 1996.
  * [Kop11] Hermann Kopetz.  Internet of Things, pages 307–323.  Springer US, Boston, MA, 2011.
  * [KVAK10] Vijay Anand Korthikanti, Mahesh Viswanathan, Gul Agha, and YoungMin Kwon.  Reasoning about mdps as transformers of probability distributions.  In Proceedings of QEST 2010, pages 199–208. IEEE Computer Society, 2010.
  * [LMMT21] Ruggero Lanotte, Massimo Merro, Andrei Munteanu, and Simone Tini.  Formal impact metrics for cyber-physical attacks.  In Proceedings of CSF 2021, pages 1–16, 2021.
  * [MN04] Oded Maler and Dejan Nickovic.  Monitoring temporal properties of continuous signals.  In Proceedings of FORMATS and FTRTFT 2004, volume 3253 of Lecture Notes in Computer Science, pages 152–166, 2004.
  * [MT16] Aditya P. Mathur and Nils Ole Tippenhauer.  Swat: a water treatment testbed for research and training on ICS security.  In Proceedings of CySWater@CPSWeek 2016, pages 31–36. IEEE Computer Society, 2016.
  * [Put05] Martin L. Puterman.  Markov Decision Processes: Discrete Stochastic Dynamic Programming.  Wiley Series in Probability and Statistics. Wiley, United States, 2005\.
  * [RKO+97] Jörg Raisch, Eberhard Klein, Siu O’Young, Christian Meder, and Alexander Itigin.  Approximating automata and discrete control for continuous systems - two examples from process control.  In Proceedings of Hybrid Systems V, 1997, volume 1567 of Lecture Notes in Computer Science, pages 279–303, 1997.
  * [RKSF13] Svetlozar T. Rachev, Lev B. Klebanov, Stoyan V. Stoyanov, and Frank J. Fabozzi.  The Methods of Distances in the Theory of Probability and Statistics.  Springer, 2013.
  * [RLSS10] Ragunathan Rajkumar, Insup Lee, Lui Sha, and John A. Stankovic.  Cyber-physical systems: the next computing revolution.  In Proceedings of DAC 2010, pages 731–736. ACM, 2010.
  * [Roy88] Halsey L. Royden.  Real Analysis.  Macmillan, New York, NY, 3rd edition, 1988.
  * [Seg95] Roberto Segala.  Modeling and verification of randomized distributed real-time systems.  PhD thesis, Massachusetts Institute of Technology, Cambridge, MA, USA, 1995.
  * [SFG+21] Bharath K. Sriperumbudur, Kenji Fukumizu, Arthur Gretton, Bernard Schölkopf, and Gert R. G. Lanckriet.  On the empirical estimation of integral probability metrics.  Electronic Journal of Statistics, 6:1550–1599, 2021.
  * [Sko56] Anatoliy V. Skorokhod.  Limit theorems for stochastic processes.  Theory Probab. Appl., 1:261–290, 1956.
  * [TBGS18] Ilya O. Tolstikhin, Olivier Bousquet, Sylvain Gelly, and Bernhard Schölkopf.  Wasserstein auto-encoders.  In Proceedings of ICLR 2018, 2018.
  * [TK10] David Thorsley and Eric Klavins.  Approximating stochastic biochemical processes with Wasserstein pseudometrics.  IET Syst. Biol., 4(3):193–211, 2010.
  * [Val74] S. S. Vallender.  Calculation of the Wasserstein distance between probability distributions on the line.  Theory Probab. Appl., 18(4):784–786, 1974.
  * [Vas69] Leonid N. Vaserstein.  Markovian processes on countable space product describing large systems of automata.  Probl. Peredachi Inf., 5(3):64–72, 1969.
  * [vB05] Franck van Breugel.  A behavioural pseudometric for metric labelled transition systems.  In Proceedings of CONCUR 2005, pages 141–155, 2005.
  * [vGSS95] Rob J. van Glabbeek, Scott A. Smolka, and Bernhard Steffen.  Reactive, generative and stratified models of probabilistic processes.  Inf. Comput., 121(1):59–80, 1995.
  * [Vil08] Cédric Villani.  Optimal transport: old and new, volume 338.  Springer, 2008.
  * [VNGV16] Csaba Virágh, Máté Nagy, Carlos Gershenson, and Gábor Vásárhelyi.  Self-organized UAV traffic in realistic environments.  In Proceedings of IROS 2016, pages 1645–1652. IEEE, 2016.