<s>[INST] Show that $Cov(AX) = A^T Cov(X) A$ For simplicity, assume random variable $X$ has mean $0$. I want to prove that $$Cov(AX) = A^T Cov(X) A$$ where A is a scalar matrix.
I tried to prove it from the definition of covariance:
$$Cov(AX) = E(AX)^T(AX) = E(X^TA^TAX)$$
I am wondering how do I bring $A$ out of the expectation. Thanks in advance.
 [/INST] Are you sure the statement you’re trying to prove isn’t $ACov(X)A^T$? Also, I believe in your definition of the covariance matrix you were applying the transpose operator to the wrong parentheses.
$$Cov(AX)=E[(AX)(AX)^T]=E(AXX^TA^T)=AE(XX^T)A^T=ACov(X)A^T$$
The statement you were trying to prove is actually incorrect. Let $x_1,x_2=\{-1,0,1\}$ be uniform random variables and $X=\begin{pmatrix} x_1 \\x_2\end{pmatrix}$ and $A= \begin{pmatrix}1 && 0 \\ 1 && 1 \end{pmatrix}$ then 
$$AX=\{
\begin{pmatrix} -1 \\ -2 \end{pmatrix},
\begin{pmatrix} -1 \\ -1 \end{pmatrix},
\begin{pmatrix} -1 \\ 0 \end{pmatrix},
\begin{pmatrix} 0 \\ -1 \end{pmatrix},
\begin{pmatrix} 0 \\ 0 \end{pmatrix},
\begin{pmatrix} 0 \\ 1 \end{pmatrix},
\begin{pmatrix} 1 \\ 0 \end{pmatrix},
\begin{pmatrix} 1 \\ 1 \end{pmatrix},
\begin{pmatrix} 1 \\ 2 \end{pmatrix}
\}$$
so $$Cov(AX)=\begin{pmatrix}2/3 && 2/3 \\ 2/3 && 4/3 \end{pmatrix}$$
but $$A^TCov(X)A=
\begin{pmatrix} 1 && 1 \\ 0 && 1 \end{pmatrix}
\begin{pmatrix} 2/3 && 0 \\ 0 && 2/3 \end{pmatrix}
\begin{pmatrix} 1 && 0 \\ 1 && 1 \end{pmatrix}=
\begin{pmatrix}4/3 && 2/3 \\ 2/3 && 2/3 \end{pmatrix}$$
</s>