I want to compare two samples with a Kolmogorov-Smirnov test. Wikipedia states the null hypothesis is rejected at $\alpha$ if:

$\sqrt( \frac{nn'}{n+n'}) D_{nn'} > K_\alpha$

where $n$ and $n'$ are the sizes of samples, D the KS-statistic and $K_\alpha$ the critical value (probably everyone here already knows). I wonder about the sample sizes: according to this formula every null hypothesis is rejected, if the samples are just large enough.

Could anybody enlighten me, what I am misunderstanding?

  • $\begingroup$ What makes you say that every null is rejected if the samples are large enough? As the sample size increases we also expect Dnn' to decrease. $\endgroup$ – Dason Nov 1 '12 at 14:49

It is correct that every result will result in rejecting the null if the sample is large enough. This is true, not just for test using KS, but for any significance test. On one level, this is very reasonable: As sample size increases, our estimates get more precise. At some point, they will get precise enough that any difference $d$ will be large enough to be different from $0$ at a level $\alpha$ .

However, it is also one of the fundamental problems with significance testing: The p-value does not (in the vast majority of cases) tell you what you want to know. That is, the question you are answering is this one:

If, in the population from which this sample was drawn there really was no effect at all, how likely is it that, in a sample of the size we have, the test statistic would be as large as the one we got, or larger?

The simple alternative is to pay more attention to effect sizes and confidence intervals. The complex alternative is to become a Bayesian. :-).

  • $\begingroup$ @PeterFlom, I agree with what you say, but I am curious from your penultimate statement: what effect size would you suggest paying attention to and what parameter(s) would you compute a confidence interval for? I would also be curious how a Bayesian would approach the comparison of 2 distributions without apriori information about what distribution or family of distributions would be used (I have heard of mixtures of Dirichlet distributions or Polya trees, but have not been able to wrap my head around those yet). $\endgroup$ – Greg Snow Nov 1 '12 at 16:30
  • $\begingroup$ Hi @GregSnow What effect size is important is really a substantive matter, rather than a statistical one. The original question doesn't give any detail on why he/she wants to do this - that would be critical. What's the substantive question? As for Bayesians - I am not one, so I will leave that answer for someone who is. $\endgroup$ – Peter Flom - Reinstate Monica Nov 1 '12 at 17:31

Your Answer

