


A common misconception about p-values







An interactive app


Let’s go through the settings!

Bildschirmfoto 2015-11-03 um 10.24.55


Bildschirmfoto 2015-11-03 um 08.30.11


Bildschirmfoto 2015-11-03 um 10.39.09


Bildschirmfoto 2015-11-03 um 10.35.09


Bildschirmfoto 2015-11-03 um 10.27.17

The formulas to compute the FDR and PPV are based on Ioannidis (2005: “Why most published research findings are false” Note that the published PDF has an error in one formula; see comments at PLOS). A related, but different approach, was proposed by David Colquhoun in his paper “An investigation of the false discovery rate and the misinterpretation of p-values” [open access]. He asks: “How should one interpret the observation of, say,  p=0.047 in a single experiment?”. The Ioannidis approach implemented in the app, in contrast, asks: “What is the FDR in a set of studies with p <= .05 and a certain power, etc.?”. Both approaches make sense, but answer different questions.

Other resources about PPV and FDR of p-values

47 thoughts on “What’s the probability that a significant p-value indicates a true effect?”

  1. Great post! There’s a typo in the tree diagram: in the calculation of the PPV, the divisor is 140, not 40.

        1. Unfortunately, that’s not possible. R-Bloggers copies the blog post when it aggregates it, but does not update it …

  2. This is nice in terms of discussing the FDR but the opening is problematic. Oakes (1986) (not Oates) wasn’t hinging the question on the FDR but on the misinterpretation that the p-value = alpha. This analysis actually answers a different question because it’s about the rate of false positives in the population of published works, not the probability I’m making a Type I error in my study, which is still alpha.

    1. I agree.
      The post was basically about how the p-value in the question is wrong which is definitely not the intended answer. (“HAHA, the question was wrong, sucker!”, doesn’t make for a good quiz)

    2. > not Oates
      Thanks for pointing out the typo.
      Concerning the Oakes question:
      What both have in common is that many researchers confuse p(Data | H0) with p(H0 | Data). Both the (wrong) answer in the question and the answer about FDR are probabilities of hypotheses, which often are tried to be inferred from a probability of data given a hypothesis.
      But you are right: the introductory example does not fit perfectly to that what follows. I will think about it.
      Actually, Oakes’ question #4 might be a better fit:
      “You can deduce the probability of the experimental hypothesis being true.”

  3. Interesting article.
    So the rule “reject the null hypothesis if p < alpha" is wrong?

    1. Well… may be. Depends on what you were taught about confidence intervals, p-value, sampling, and experiments. Strictly speaking, as I was taught, the CI and p-value are referenced to the experiment’s design: out of 100 identical experiments, how many will capture the true value in the CI? In that view, neither the p-value nor CI refer to any single experiment/observational study.
      An interesting quote from the Wiki ( https://en.wikipedia.org/wiki/Type_I_and_type_II_errors#Etymology )
      “… as Florence Nightingale David remarked, “it is necessary to remember the adjective ‘random’ [in the term ‘random sample’] should apply to the method of drawing the sample and not to the sample itself”. ”
      Whether the Bayesians have it better is another topic.

  4. Nice post and nice illustration. In terms of the Nature Methods article linked above, it’s a bit of a shame that the caption to Figure 1 mentions underpowered studies. If the majority of effects we are testing don’t really exist – say only 5% of them – then even with a statistical power of 0.95, the FDR is 50%!
    So let’s say we have a huge (and ever increasing number) of researchers desperately trying to get that cool effect so that they can get published in Science/Nature. And let’s say that the underlying, *mechanistic* theory behind their hypothesised effect is, shall we say, very shaky (i.e. they *think*, based on some wordy theory, that there might be an effect). How many of those researchers will be searching for an effect that really exists? I think 5% is quite within the bounds of reason. In which case, even amongst studies with a very high statistical power, there’s a 50% chance that the exciting, remarkable effect that got published in Science/Nature is false.
    Increasing statistical power (while always a good idea) won’t help in this case. What will help are: i) better understanding of mechanisms that allow for more realistic hypotheses, ii) use of better statistics than standard NHST, and iii) replication (in the case I mention, you would need to show 5 successive replications to achieve an FDR < 0.05).

  5. Good post, but this seems unlikely: “On the other hand, the rate can definitely drop close to zero. For example, in pharmaceutical drug development “only one in every 5,000 compounds that makes it through lead development to the stage of pre-clinical development becomes an approved drug” (Wikipedia). Here, only 0.02% of all investigated hypotheses are true.” Drug approval depends on factors other than whether there is a true effect (cost, size of effect and patient safety being good examples).

    1. Right – many of these 5000 compounds are not actual false-positives, but drop out for other reasons. But even if the rate of working compounds is 100x higher, the a priori rate still is at a meager 2%. (“meager” WRT to the consequences for the PPV).
      Hence: Maybe these specific numbers are exaggerated, but I think one can still make the point that there are many reasonable scientific scenarios where the a priori rate of H1 being true gets close to zero.
      (Ioannidis, 2005, for example sets the rate for “Discovery-oriented exploratory research with massive testing” at 0.01%)

  6. Very nice post. What I take away is that what calculation you make depends precisey on what question you want to answer. Oakes’s question is, in this context, distressingly imprecise. So, as I recall from my (distant) statstics course is that the the usual (what you call wrong) interpretation of the p-value is something like: the chance of rejecting the null when the null is true (or sometimes, type I error). So, the entire right branch of your decision tree is chopped off, and you get the same answer as the textbook answer. If people were interpreting Oakes’s question this way, and I think that is one perfectly reasonable interpretation of the vague question, then they were right, not wrong. You claim this interpretation is essentially trivial, but again it depends on the question, as psyachoskeptic and others have already noted. I am not a statistician, but I think the question is a reasonable one.
    That said, it is all to easy to let that question slip into: Is the null true? or Is the null false? Your post is a great illustration of the pitfalls of making this very common slip, and has sound suggestions on how to deal with the problem.
    Thanks again for the post.

  7. A late reply and probably missing a point: It is a great post and highly instructive. However, it hinges on having a share of real effects in 1000 studies. When I read the initial quote, I was thinking of a single study, not of 1000, and a single relationship for which the effect is either real or not. If you do 1000 studies on how listening to country music is related to suicide (came across this thanks to Lakens), then the effect of country music is real or not (probably not) and your tree diagram does not apply. If country music has no effect different from 0, then all significant results are false positives. You can apply the diagram, of course, but you end with 100% and 0%. It would be different if you were having 1000 studies, some of which test country music and others test different music genres for which there might be an effect. Is this correct?

  8. Hello. This app seems to be amazing. However, I am having troubles to make it work. I’ve tried 4 different browsers in 2 different laptops (Mac and PC) and it doesn’t seem to work for me (the screen is in grey colour and shifting the sliders doesn’t change anything). Would have any advice on what I could do to make it work, please? This would be of a massive help.
    Thanks very much,
    All the best,

    1. Some universities block port 3838, which is needed for Shiny apps. Did you also try it from home?

Leave a Reply to Thom Cancel reply

Your email address will not be published.
