# The $\chi^2$ Distribution

## $\chi^2$ Test Statistic

If we make $n$ ranom samples (observations) from Gaussian (Normal) distributions with known means, $\mu_i$, and known variances, $\sigma_i^2$, it is seen that the total squared deviation,

$$
\chi^2 = \sum_{i=1}^{n} \left(\frac{x_i - \mu_i}{\sigma_i}\right)^2\,,
$$

follows a $\chi^2$ distribution with $n$ degrees of freedom.

## Probability Distribution Function

The $\chi^2$ probability distribution function for $k$ degrees of freedom (the number of parameters that are allowed to vary) is given by

$$
f\left(\chi^2\,;k\right) = \frac{\displaystyle 1}{\displaystyle 2^{k/2} \,\Gamma\left(k\,/2\right)}\, \chi^{k-2}\,e^{-\chi^2/2}\,,
$$

where if there are no constrained variables the number of degrees of freedom, $k$, is equal to the number of observations, $k=n$. The p.d.f. is often abbreviated in notation from $f\left(\chi^2\,;k\right)$ to $\chi^2_k$.

A reminder that for integer values of $k$, the Gamma function is $\Gamma\left(k\right) = \left(k-1\right)!$, and that $\Gamma\left(x+1\right) = x\Gamma\left(x\right)$, and $\Gamma\left(1/2\right) = \sqrt{\pi}$.

## Mean

Letting $\chi^2=z$, and noting that the form of the Gamma function is

$$
\Gamma\left(z\right) = \int\limits_{0}^{\infty} x^{z-1}\,e^{-x}\,dx,
$$

it is seen that the mean of the $\chi^2$ distribution $f\left(\chi^2 ; k\right)$ is

$$
\begin{align}
\mu &= \textrm{E}\left[z\right] = \displaystyle\int\limits_{0}^{\infty} z\, \frac{\displaystyle 1}{\displaystyle 2^{k/2} \,\Gamma\left(k\,/2\right)}\, z^{k/2-1}\,e^{-z\,/2}\,dz \\
    &= \displaystyle \frac{\displaystyle 1}{\displaystyle \Gamma\left(k\,/2\right)} \int\limits_{0}^{\infty} \left(\frac{z}{2}\right)^{k/2}\,e^{-z\,/2}\,dz = \displaystyle \frac{\displaystyle 1}{\displaystyle \Gamma\left(k\,/2\right)} \int\limits_{0}^{\infty} x^{k/2}\,e^{-x}\,2 \,dx \\
    &= \displaystyle \frac{\displaystyle 2 \,\Gamma\left(k\,/2 + 1\right)}{\displaystyle \Gamma\left(k\,/2\right)} \\
    &= \displaystyle 2 \frac{k}{2} \frac{\displaystyle \Gamma\left(k\,/2\right)}{\displaystyle \Gamma\left(k\,/2\right)} \\
    &= k.
\end{align}
$$

## Variance

Likewise, the variance is

$$
\begin{align}
\textrm{Var}\left[z\right] &= \textrm{E}\left[\left(z-\textrm{E}\left[z\right]\right)^2\right] = \displaystyle\int\limits_{0}^{\infty} \left(z - k\right)^2\, \frac{\displaystyle 1}{\displaystyle 2^{k/2} \,\Gamma\left(k\,/2\right)}\, z^{k/2-1}\,e^{-z\,/2}\,dz \\
    &= \displaystyle\int\limits_{0}^{\infty} z^2\, f\left(z \,; k\right)\,dz - 2k\int\limits_{0}^{\infty} z\,\,f\left(z \,; k\right)\,dz + k^2\int\limits_{0}^{\infty} f\left(z \,; k\right)\,dz \\
    &= \displaystyle\int\limits_{0}^{\infty} z^2 \frac{\displaystyle 1}{\displaystyle 2^{k/2} \,\Gamma\left(k\,/2\right)}\, z^{k/2-1}\,e^{-z\,/2}\,dz - 2k^2 + k^2\\
    &= \displaystyle\int\limits_{0}^{\infty} \frac{\displaystyle 1}{\displaystyle 2^{k/2} \,\Gamma\left(k\,/2\right)}\, z^{k/2+1}\,e^{-z\,/2}\,dz - k^2\\
    &= \frac{\displaystyle 2}{\displaystyle \Gamma\left(k\,/2\right)} \displaystyle\int\limits_{0}^{\infty} \left(\frac{z}{2}\right)^{k/2+1}\,e^{-z\,/2}\,dz - k^2 = \frac{\displaystyle 2}{\displaystyle \Gamma\left(k\,/2\right)} \displaystyle\int\limits_{0}^{\infty} x^{k/2+1}\,e^{-x}\,2\,dx - k^2 \\
    &= \displaystyle \frac{\displaystyle 4 \,\Gamma\left(k\,/2 + 2\right)}{\displaystyle \Gamma\left(k\,/2\right)} - k^2 \\
    &= \displaystyle 4 \left(\frac{k}{2} + 1\right) \frac{\displaystyle \Gamma\left(k\,/2 + 1\right)}{\displaystyle \Gamma\left(k\,/2\right)} - k^2 \\
    &= \displaystyle 4 \left(\frac{k}{2} + 1\right) \frac{k}{2} - k^2 \\
    &= k^2 + 2k - k^2 \\
    &= 2k,
\end{align}
$$

such that the standard deviation is

$$
\sigma = \sqrt{2k}\,.
$$

Given this information we now plot the $\chi^2$ p.d.f. with various numbers of degrees of freedom to visualize how the distribution's behaviour


```python
import numpy as np
import scipy.stats as stats

import matplotlib.pyplot as plt
```


```python
# Plot the chi^2 distribution
x = np.linspace(0.0, 10.0, num=1000)

[plt.plot(x, stats.chi2.pdf(x, df=ndf), label=fr"$k = ${ndf}") for ndf in range(1, 7)]

plt.ylim(-0.01, 0.5)

plt.xlabel(r"$x=\chi^2$")
plt.ylabel(r"$f\left(x;k\right)$")
plt.title(r"$\chi^2$ distribution for various degrees of freedom")

plt.legend(loc="best")

plt.show();
```

## Cumulative Distribution Function

The cumulative distribution function (CDF) for the $\chi^2$ distribution is (letting $z=\chi^2$)

$$
\begin{split}
F_{\chi^2}\left(x\,; k\right) &= \int\limits_{0}^{x} f_{\chi^2}\left(z\,; k\right) \,dz \\
    &= \int\limits_{0}^{x} \frac{\displaystyle 1}{\displaystyle 2^{k/2} \,\Gamma\left(k\,/2\right)}\, z^{k/2-1}\,e^{-z/2} \,dz \\
    &= \int\limits_{0}^{x} \frac{\displaystyle 1}{\displaystyle 2 \,\Gamma\left(k\,/2\right)}\, \left(\frac{z}{2}\right)^{k/2-1}\,e^{-z/2} \,dz = \frac{1}{\displaystyle 2 \,\Gamma\left(k\,/2\right)}\int\limits_{0}^{x/2} t^{k/2-1}\,e^{-t} \,2\,dt \\
    &= \frac{1}{\displaystyle \Gamma\left(k\,/2\right)}\int\limits_{0}^{x/2} t^{k/2-1}\,e^{-t} \,dt
\end{split}
$$

Noting the form of the [lower incomplete gamma function](https://en.wikipedia.org/wiki/Incomplete_gamma_function) is

$$
\gamma\left(s,x\right) = \int\limits_{0}^{x} t^{s-1}\,e^{-t} \,dt\,,
$$

and the form of the [regularized Gamma function](https://en.wikipedia.org/wiki/Incomplete_gamma_function#Regularized_Gamma_functions_and_Poisson_random_variables) is

$$
P\left(s,x\right) = \frac{\gamma\left(s,x\right)}{\Gamma\left(s\right)}\,,
$$

it is seen that

$$
\begin{split}
F_{\chi^2}\left(x\,; k\right) &= \frac{1}{\displaystyle \Gamma\left(k\,/2\right)}\int\limits_{0}^{x/2} t^{k/2-1}\,e^{-t} \,dt \\
    &= \frac{\displaystyle \gamma\left(\frac{k}{2},\frac{x}{2}\right)}{\displaystyle \Gamma\left(\frac{k}{2}\right)} \\
    &= P\left(\frac{k}{2},\frac{x}{2}\right)\,.
\end{split}
$$

Thus, it is seen that the compliment to the CDF (the complementary cumulative distribution function (CCDF)),

$$
\bar{F}_{\chi^2}\left(x\,; k\right) = 1-F_{\chi^2}\left(x\,; k\right),
$$

represents a one-sided (one-tailed) $p$-value for observing a $\chi^2$ given a model &mdash; that is, the probability to observe a $\chi^2$ value greater than or equal to that which was observed.


```python
def chi2_ccdf(x, df):
    """The complementary cumulative distribution function

    Args:
        x: the value of chi^2
        df: the number of degrees of freedom

    Returns:
        1 - the cumulative distribution function
    """
    return 1.0 - stats.chi2.cdf(x=x, df=df)
```


```python
x = np.linspace(0.0, 10.0, num=1000)
fig, axes = plt.subplots(nrows=1, ncols=2, figsize=(14, 4.5))

for ndf in range(1, 7):
    axes[0].plot(x, stats.chi2.cdf(x, df=ndf), label=fr"$k = ${ndf}")
    axes[1].plot(x, chi2_ccdf(x, df=ndf), label=fr"$k = ${ndf}")

axes[0].set_xlabel(r"$x=\chi^2$")
axes[0].set_ylabel(r"$F\left(x;k\right)$")
axes[0].set_title(r"$\chi^2$ CDF for various degrees of freedom")

axes[0].legend(loc="best")

axes[1].set_xlabel(r"$x=\chi^2$")
axes[1].set_ylabel(r"$\bar{F}\left(x;k\right) = p$-value")
axes[1].set_title(r"$\chi^2$ CCDF ($p$-value) for various degrees of freedom")

axes[1].legend(loc="best")

plt.show();
```

## Binned $\chi^2$ per Degree of Freedom

TODO

## References

- \[1\] G. Cowan, _Statistical Data Analysis_, Oxford University Press, 1998
- \[2\] G. Cowan, "Goodness of fit and Wilk's theorem", Notes, 2013

4d167b6ca37e89b420b3ca15def2abf0006d0085
ipynb
Jupyter Notebook
book/notebooks/Introductory/Chi-Squared-Distribution.ipynb
matthewfeickert/Statistics-Notes
088181920b0f560fdd2ed593d3653f67baa56190
MIT
2018-02-15T15:22:15.000Z
2022-02-18T07:28:57.000Z
book/notebooks/Introductory/Chi-Squared-Distribution.ipynb
matthewfeickert/Statistics-Notes
088181920b0f560fdd2ed593d3653f67baa56190
MIT
2018-05-08T22:51:03.000Z
2021-12-08T03:58:30.000Z
book/notebooks/Introductory/Chi-Squared-Distribution.ipynb
matthewfeickert/Statistics-Notes
088181920b0f560fdd2ed593d3653f67baa56190
MIT
2018-10-24T17:15:44.000Z
2020-04-13T00:27:06.000Z
Qwen/Qwen-72B
1. YES
2. YES