Imagine a 6-sided die. I roll it N times and calculate the Chi-Square statistic on the 6-entry table of frequencies of these rolls.

In my program, I calculate the chi-square statistic as follows:

$\sum_i (O_i-E_i)^2/E_i^2$

The distribution of this statistic is supposed to follow a Chi-Square distribution with DOF 5. The theoretical distribution of the above statistic, as far as I can tell from the literature on using Pearson's Chi-Square Test to test fit to a distribution, is not sensitive to the number of samples/rolls (number N).

However, in practice, it is sensitive to N.

Therefore, the Chi-Square Distribution cannot model this statistic for more than one particular value of N (but we do not specify N when we specify the distribution, only DOF which is related to the number of free variables specifying frequencies).

Yes, I do not know deeply the theoretical foundation of Chi-Square so I am looking for a way of understanding this seeming paradox.

I can put this question another way.

In this code, I have a simulation that I run M times (fixed high M e.g. 10,000). In each simulation I simulate rolling a fair die N times to get N samples and calculate the Chi-Square statistic. For each different choice of N I get a different histogram of the Chi Square statistic (similar shape, different scale). I also sample from the theoretical Chi-Square distribution with DOF 5 (M times) and get the same shape but different values. The 95th percentile of the samples from the theoretical distribution is around 11. From the empirical distributions, for N=200 it is around 1.5 and for N=5000 it is around 1.2 (this decreasing of the threshold would seem appropriate as I am increasing the number of samples from which my statistic is calculated, but certainly is not approaching the theoretical value). Obviously, if I were to use the theoretical distribution against an experiment with N=200 I would be overestimating the threshold that would be required to make a type-1 error 5% of the time when the die is fair (recalling the aim is to check fit to distribution - in this case a uniform). As far as I can tell, none of this converges to the theoretical distribution as either N or M increase.

Where is my fundamental mistake?

Thank you in advance for your feedback.

There was no fundamental mistake. Thanks to Glen_b's own simulation which gave theoretically appropriate results, I could see that there must have been a problem with the code, which there was. The primary problem was not the formula. Simply, the code was sampling from a 5 sided die where it should have been from a 6 sided die.

  • $\begingroup$ 1. The distribution of the chi-square statistic does depend on N. Have you seen the widely-used "rule of thumb" that the expected value in each cell should be greater than 5? (or any of a number of similar rules) ... Such rules exist because the chi-squared approximation for the distribution of the statistic is not very good when there are small expected values, but it's better as $N$ increases 2. However, if I follow what you're saying correctly you have some other issue that the one I am describing. 3. It's impossible to tell from the infor in the Q what it is you're doing wrong; ... $\endgroup$ – Glen_b May 9 '18 at 15:47
  • $\begingroup$ ... and your link doesn't work -- it wants me to log in. In any case, could you perhaps explain what you're trying to do? $\endgroup$ – Glen_b May 9 '18 at 15:57
  • $\begingroup$ Sorry about the link. Fixed that. From my experiments, as I increase N, the 95% threshold actually goes down (from 1.5 when N=200) but the theoretical threshold for 5 DOF is 11.1 (p=0.05). $\endgroup$ – D.J.Duff May 9 '18 at 16:32
  • $\begingroup$ But I take your point that the threshold calculated from the empirical distribution does seem to converge as N grows. Thank you. $\endgroup$ – D.J.Duff May 9 '18 at 16:33
  • $\begingroup$ In the die example, if I set N to 6, that is I calculate Chi-Square from 6 throws, then the 95th percentile is generally about 12 which is the closest I got to the theoretical threshold for p=0.05. Could it be that when using the Chi-Square distribution for Pearson's Chi-Square test, we assume the number of bins and the sum of the frequencies in the bins to be roughly equal? Because if I increase N, as I wrote, the real threshold gets too low to be comparable to the theoretical distribution. $\endgroup$ – D.J.Duff May 9 '18 at 19:43

Your formula for the chi-squared statistic is wrong.

It should be $\sum_i (O_i-E_i)^2/E_i$.

When you see surprising or unexpected behavior, it's the obvious thing to check.

With a correct implementation of the statistic you should see a progression something like this:

histograms of chi-squared statistic for the described experiment and the chi-squared density with five degrees of freedom

Here we see histograms for your experiment compared to the $\chi^2_5$ density with N = 25, 50, 100, 200

As you see, things are much as one might expect -- the $\chi^2$ approximation is rough at N=25 ($E_i$ approximately 4 in each cell) and considerably better at $N=200$ (where $E_i$ is approximately 33 in each cell).

In case anyone feels the need to see it, here's my code (done in R) for N=200:

X2.200 = replicate(10000,

This is not especially nicely organized, but okay for a quick simulation (I actually had this as a single line but it's a couple of characters too long for the window).

  • $\begingroup$ Thank you for editing my question to add the formula on the basis of my code, and thank you for pointing that out. I do appreciate your help. Unfortunately, that was only a change that I made to try and achieve convergence to the theoretical. Reverting my code to the non-squared denominator I get a growth in the 95th percentile of chi-square statistic, way beyond the theoretical (N=7 gives me 11, N=200 gives 51, N=5000 gives 1011). $\endgroup$ – D.J.Duff May 10 '18 at 3:45
  • $\begingroup$ So I'll update the formula in the question. Thank you for adding it. $\endgroup$ – D.J.Duff May 10 '18 at 3:46
  • $\begingroup$ If you change the formula in the question, it won't correspond to your code. You should fix your code instead and see if this fixes your problem $\endgroup$ – Glen_b May 10 '18 at 4:24
  • $\begingroup$ Thanks Glen. I changed the code before I changed the question. The link should reflect the fixed code. $\endgroup$ – D.J.Duff May 10 '18 at 4:27
  • $\begingroup$ The only change in the code was to the was to the Chi-Square formula. $\endgroup$ – D.J.Duff May 10 '18 at 4:28

Your Answer

