Cochran's theorem

In statistics, Cochran's theorem, devised by William G. Cochran, is a theorem used to justify results relating to the probability distributions of statistics that are used in the analysis of variance.
Statement.
Let "U"1, ..., "U""N" be i.i.d. standard normally distributed random variables, and formula_1. Let formula_2be symmetric matrices. Define "r""i" to be the rank of formula_3. Define formula_4, so that the "Q"i are quadratic forms. Further assume formula_5. 
Cochran's theorem states that the following are equivalent: 
Often it's stated as formula_7, where formula_8 is idempotent, and formula_9 is replaced by formula_10. But after an orthogonal transform, formula_11, and so we reduce to the above theorem.
Proof.
Claim: Let formula_12 be a standard Gaussian in formula_13, then for any symmetric matrices formula_14, if formula_15 and formula_16 have the same distribution, then formula_14 have the same eigenvalues (up to multiplicity).
Proof: Let the eigenvalues of formula_18 be formula_19, then calculate the characteristic function of formula_15. It comes out to be 
formula_21
For formula_15 and formula_16 to be equal, their characteristic functions must be equal, so formula_14 have the same eigenvalues (up to multiplicity).
Claim: formula_26.
Proof: formula_27. Since formula_28 is symmetric, and formula_29, by the previous claim, formula_28 has the same eigenvalues as 0.
Lemma: If formula_31, all formula_32 symmetric, and have eigenvalues 0, 1, then they are simultaneously diagonalizable.
Fix i, and consider the eigenvectors v of formula_33 such that formula_34. Then we have formula_35, so all formula_36. Thus we obtain a split of formula_37 into formula_38, such that V is the 1-eigenspace of formula_33, and in the 0-eigenspaces of all other formula_40. Now induct by moving into formula_41.
Case: All formula_42 are independent
Fix some formula_43, define formula_44, and diagonalize formula_45 by an orthogonal transform formula_46. Then consider formula_47. It is diagonalized as well.
Let formula_48, then it is also standard Gaussian. Then we have 
formula_49
Inspect their diagonal entries, to see that formula_50 implies that their nonzero diagonal entries are disjoint.
Thus all eigenvalues of formula_45 are 0, 1, so formula_42 is a formula_53 dist with formula_54 degrees of freedom.
Case: Each formula_42 is a formula_56 distribution.
Fix any formula_43, diagonalize it by orthogonal transform formula_46, and reindex, so that formula_59. Then formula_60 for some formula_61, a spherical rotation of formula_62.
Since formula_63, we get all formula_64. So all formula_65, and have eigenvalues formula_66.
So diagonalize them simultaneously, add them up, to find formula_9.
Case: formula_6
We first show that the matrices "B"("i") can be simultaneously diagonalized by an orthogonal matrix and that their non-zero eigenvalues are all equal to +1. Once that's shown, take this orthogonal transform to this simultaneous eigenbasis, in which the random vector formula_69 becomes formula_70, but all formula_71 are still independent and standard Gaussian. Then the result follows.
Each of the matrices "B"("i") has rank "r""i" and thus "r""i" non-zero eigenvalues. For each "i", the sum formula_72 has at most rank formula_73. Since formula_74, it follows that "C"("i") has exactly rank "N" − "r""i".
Therefore "B"("i") and "C"("i") can be simultaneously diagonalized. This can be shown by first diagonalizing "B"("i"), by the spectral theorem. In this basis, it is of the form:
Thus the lower formula_76 rows are zero. Since formula_77, it follows that these rows in "C"("i") in this basis contain a right block which is a formula_78 unit matrix, with zeros in the rest of these rows. But since "C"("i") has rank "N" − "r""i", it must be zero elsewhere. Thus it is diagonal in this basis as well. It follows that all the non-zero eigenvalues of both "B"("i") and "C"("i") are +1. This argument applies for all "i", thus all "B"("i") are positive semidefinite.
Moreover, the above analysis can be repeated in the diagonal basis for formula_79. In this basis formula_80 is the identity of an formula_81 vector space, so it follows that both "B"(2) and formula_82 are simultaneously diagonalizable in this vector space (and hence also together with "B"(1)). By iteration it follows that all "B"-s are simultaneously diagonalizable.
Thus there exists an orthogonal matrix formula_83 such that for all formula_43, formula_85 is diagonal, where any entry formula_86 with indices formula_87, formula_88, is equal to 1, while any entry with other indices is equal to 0.
Examples.
Sample mean and sample variance.
If "X"1, ..., "X""n" are independent normally distributed random variables with mean "μ" and standard deviation "σ" then
is standard normal for each "i". Note that the total "Q" is equal to sum of squared "U"s as shown here:
which stems from the original assumption that formula_91.
So instead we will calculate this quantity and later separate it into "Q""i"'s. It is possible to write
(here formula_93 is the sample mean). To see this identity, multiply throughout by formula_94 and note that
and expand to give
The third term is zero because it is equal to a constant times
and the second term has just "n" identical terms added together. Thus
and hence
Now formula_100 with formula_101 the matrix of ones which has rank 1. In turn formula_102 given that formula_103. This expression can be also obtained by expanding formula_104 in matrix notation. It can be shown that the rank of formula_105 is formula_106 as the addition of all its rows is equal to zero. Thus the conditions for Cochran's theorem are met.
Cochran's theorem then states that "Q"1 and "Q"2 are independent, with chi-squared distributions with "n" − 1 and 1 degree of freedom respectively. This shows that the sample mean and sample variance are independent. This can also be shown by Basu's theorem, and in fact this property "characterizes" the normal distribution – for no other distribution are the sample mean and sample variance independent.
Distributions.
The result for the distributions is written symbolically as
Both these random variables are proportional to the true but unknown variance "σ"2. Thus their ratio does not depend on "σ"2 and, because they are statistically independent. The distribution of their ratio is given by
where "F"1,"n" − 1 is the F-distribution with 1 and "n" − 1 degrees of freedom (see also Student's t-distribution). The final step here is effectively the definition of a random variable having the F-distribution.
Estimation of variance.
To estimate the variance "σ"2, one estimator that is sometimes used is the maximum likelihood estimator of the variance of a normal distribution
Cochran's theorem shows that
and the properties of the chi-squared distribution show that
Alternative formulation.
The following version is often seen when considering linear regression. Suppose that formula_113 is a standard multivariate normal random vector (here formula_114 denotes the "n"-by-"n" identity matrix), and if formula_115 are all "n"-by-"n" symmetric matrices with formula_116. Then, on defining formula_117, any one of the following conditions implies the other two: