Problematic Neuropeptides And Statistics (PNAS)



Back in May I discussed a paper published in PNAS which, I claimed, was using scientific terminology in a sloppy way. The authors, Pearce et al., used the word “neuropeptides” to refer to six molecules, but three of them weren’t neuropeptides at all. The authors acknowledged this minor error and issued a correction.




Of these tests, 44 (4.7%) were statistically significant at the p<0.05 level, and Jern et al. point out that we’d expect a 5% positive rate by chance. Jern et al. go on to note that Pearce et al. did not clarify how many of the results survived multiple comparison correction:

Pearce et al. allude to having corrected for multiple testing using the mperm option in Plink. However, they do not report corrected P values, but instead report in the SI Appendix that ‘genotypic model significance levels did not always survive correction for multiple tests using mperm.’ It is unclear how many tests, if any, did survive multiple testing correction.


Pearce_fig_2Pearce et al. said of this table that


I think this analysis is badly flawed. First off, I can’t see how the Fig. 2 data give a chi-squared (χ2) statistic of 51.7, p<0.0001. As far as I can see, the statistics are a less impressive χ2 = 20.4, p=0.013. Also, as Jern et al. point out, there are some discrepencies between Fig. 2 and the raw data, but these don’t change the statistics much.

The numbers don’t really matter, however, because the χ2 test is invalid. The main problem is that the observations (i.e. significant SNP-behavior associations) are not independent.

Suppose that we observe a significant association between, say, an oxytocin SNP and a dyadic relationships measure. And say we observe a second association between a different oxytocin SNP and the same measure. This makes two observations to go into the same cell of Fig. 2, but they’re not independent observations, if the two oxytocin SNPs are correlated with each other.



The LD problem alone makes Pearce et al.’s χ2 analysis flawed, because χ2 assumes independence of observations. There are also significant correlations among the eight different behavioural tasks, which makes matters worse.



What’s sauce for the goose is sauce for the gander, and if LD makes mperm invalid, it certainly does so for χ2.

Finally, to rub salt into the wound, Jern et al. conclude their letter by saying they failed to replicate many of Pearce et al.’s findings:


Dunbar in his reply says that:




Source link