If a PDF is re-constructed as following $p(x)=\sum_{k=1}^{K}\pi_k p(x|k),$ where $p(x|k)$ has mean $\mu_k$ and covariance matrix $\Sigma_k$ , compute the mean and covariance of the new mixture distribution.

Finding the mean is trivial , so say that I've found the mean and denote it as $mean=\left[ \begin{array}{cl} mean_{1} \\ ... \\ mean_{D} \end{array}\right]$

I'm quite confused about the dimension of covariance matrix, say I have $D$ dimension for each observation $x=\left[ \begin{array}{cl} x_{1} \\ ... \\ x_{D} \end{array}\right]$, then dimension of covariance matris should be $D \times D,$ but what 's the concrete element in position $<i,j>$ ?

  • $\begingroup$ By far what I got is $\Sigma=(\sum_{k=1}^{K} \pi_k \Sigma_k) - mean \cdot mean^T + [\sum_{k=1}^K \pi_k \mu_{ki} \mu_{kj}]$, where (i,j) means the position in a matrix $\endgroup$ – genxium May 20 '12 at 19:58
  • $\begingroup$ Basically the same question (and answer) here: math.stackexchange.com/questions/195911/… $\endgroup$ – Yibo Yang Mar 4 at 6:20

I think what you mean is this. There is a random variable $Y$ with possible values $1, \ldots, K$ and corresponding probabilities $\pi_k$. Your random variable $X$ is a $d$-dimensional vector with conditional expectations $\text{E}[X | Y=k] = \mu_k$ (a $d$-dimensional vector) and conditional covariances $\text{Cov}(X | Y=k) = C_k$ (a $d \times d$ matrix). I'd rather use this instead of $\Sigma_k$ because I want to use $\sum_k$ for summation over $k$. Thus $\text{Cov}(X_i,X_j | Y = k) = \text{E}(X_i X_j | Y = k) - \text{E}(X_i | Y=k) \text{E}(X_j | Y = k) = (C_k)_{ij}$. Now the unconditional expectation is $\text{E}(X) = \sum_k E(X| Y=k) \pi_k = \sum_k \mu_k \pi_k$. On the other hand, $$\text{E}(X_i X_j) = \sum_k \text{E}(X_i X_j | Y=k) \pi_k = \sum_k \left((C_k)_{ij} + (\mu_k)_i (\mu_k)_j\right) \pi_k$$ so $$ \eqalign{\text{Cov}(X_i,X_j) &= \text{E}(X_i X_j) - E(X_i) E(X_j)\cr & = \sum_k \left((C_k)_{ij} + (\mu_k)_i (\mu_k)_j \right) \pi_k - \sum_k (\mu_k)_i \pi_k \sum_{k'} (\mu_k')_j \pi_{k'}\cr} $$ which can be expressed as $$ \text{Cov}(X_i,X_j) = \text{E}\left[{\text{Cov}(X_i,X_j | Y)}\right] + \text{Cov}(E(X_i|Y), E(X_j | Y))$$ the "law of total covariance" formula: see http://en.wikipedia.org/wiki/Law_of_total_covariance These entries form a matrix $$ C = \sum_k C_k \pi_k + \sum_k \mu_k^T \mu_k \pi_k - \mu^T \mu $$ where $\mu = E(X)$.

  • $\begingroup$ Thanks for you answer buddy, but I can't tell if this is right or wrong because I actually got the same reasoning, as my exams are just finished yesterday, I'll consult my professor for this problem soon, hope he will give me the same result :) $\endgroup$ – genxium May 26 '12 at 18:26

Your Answer

