If nonparametric tests are assumed to have less power than their parametric alternatives, does this imply that if any parametric test does not reject null, then its nonparametric alternative does not reject null too? How can this change if assumptions of parametric test are not met and the test is used anyway?

  • 4
    $\begingroup$ (a) if test A has lower power than test B under some assumed circumstance, that doesn't imply that the cases that A will reject are a subset of those in B (in fact that's not possible at a fixed significance level); they reject different (but perhaps heavily overlapping) portions of the sample space -- there are always cases each will reject that the other will not. (b) If the assumptions of the parametric test are not met (are they ever actually met?), then parametric tests may have relatively poor power (e.g. t-test vs Mann-Whitney under heavy tails) $\endgroup$ – Glen_b Aug 29 '13 at 23:50

If a parametric test fails to reject the null hypothesis then its nonparametric equivalent can definitely still reject the null hypothesis. Like @John said, this usually occurs when assumptions that would warrant use of the parametric test are violated. For example, if we compare the two-sample t-test with the Wilcoxon rank sum test then we can get this situation to happen if we include outliers in our data (with outliers we should not use the two sample-test).

#Test Data
x = c(-100,-100,rnorm(1000,0.5,1),100,100)
y = rnorm(1000,0.6,1)

#Two-Sample t-Test

#Wilcoxon Rank Sum Test

The results of running the test:

> t.test(x,y,var.equal=TRUE)

    Two Sample t-test

data:  x and y 
t = -1.0178, df = 2002, p-value = 0.3089
alternative hypothesis: true difference in means is not equal to 0 
95 percent confidence interval:
 -0.6093287  0.1929563 
sample estimates:
mean of x mean of y 
0.4295556 0.6377417 

> wilcox.test(x,y)

    Wilcoxon rank sum test with continuity correction

data:  x and y 
W = 443175, p-value = 5.578e-06
alternative hypothesis: true location shift is not equal to 0 


While parametric tests can be more powerful that's not always the case. When it's not the case then it's usually in situations where you shouldn't be running the parametric tests.

But, even if you're collecting decent sized samples from normal distributions with equal variance where the parametric test has higher power, it doesn't guarantee that for any particular experiment a non-significant parametric test means a non-significant nonparametric test. Here's a simulation that just uses random sampling from normal distributions and finds that about 1.8% of the time when p > 0.05 for a t-test that p < 0.05 for a Wilcoxon test.

nsim <- 10000
n <- 50
cohensD <- 0.2
Y <- replicate(nsim, {
    y1 <- rnorm(n, 0, 1); y2 <- rnorm(n, cohensD, 1)
    tt <- t.test(y1, y2, var.equal = TRUE)
    wt <- wilcox.test(y1, y2)
    c(tt$p.value, wt$p.value)})
sum(Y[1,] > 0.05 & Y[2,] < 0.05) / nsim

You might note that, in this simulation, the power of the parametric test is greater than the nonparametric test (although, they are similar).

sum(Y[1,] < 0.05) / nsim #t-test power
sum(Y[2,] < 0.05) / nsim #wilcox.test power

But, as is shown above, that does not mean the that in all cases where the parametric test fails to find an effect that the nonparametric test fails as well.

You can play with this simulation. Make n quite large, say 1000, and make the effect size much smaller, say 0.02 (you need low power to have lots of samples where the test fails). You can be pretty much guaranteed with an n of 1000 that none of the samples would be rejected for non-normality (by inspection, not a stupid test) or have suspicious outliers. Yet still, some of the parametric tests come out non significant while the nonparametric tests are significant.

You also might want to look at Hunter & May (1993).

Hunter, M. A., & May, R. B. (1993). Some myths concerning parametric and nonparametric tests. Canadian Psychology, 34(4), 384-389.


Your Answer

