<s>[INST] Expected value of the distance of sample average from the overall average Let $\mathbf{y}_1, \mathbf{y}_2, \dots, \mathbf{y}_N$ be a sequence of $N$ vectors in $\mathbb{R}^d$ and $\bar{\mathbf{y}}_N$ be the overall average, i.e.,
$$
\bar{\mathbf{y}}_N=\frac{1}{N}\sum_{i=1}^{N}\mathbf{y}_i.
$$
Also, let $\bar{\mathbf{y}}$ be any sample average of size $n$ such that $\bar{\mathbf{y}}=\frac{1}{n}\sum_{i \in A}\mathbf{y}_i$ where $A$ is a random set such that $A \subset \{1, \dots, N\}$ and $|A|=n<N$. Note that elements in $A$ are drawn without replacement.
Can we show
$$
\mathbb{E}_A[\|\bar{\mathbf{y}}-\bar{\mathbf{y}}_N\|^2] = \frac{N-n}{n}\frac{1}{N}e
$$
where $e=\frac{1}{N-1}\sum_{i=1}^N\|\mathbf{y}_i-\bar{\mathbf{y}}_N\|^2$?
My try:
The above problem is solved in Sampling: design and analysis page 45-46. I do not know how to do it efficiently by defining $Z_i$'s as random variables that take only zero or one like the scalar case. I also think maybe it is better to write $\mathbb{E}_A[\|\bar{\mathbf{y}}-\bar{\mathbf{y}}_N\|^2]$ as $\mathbb{E}_A[ \langle \bar{\mathbf{y}}-\bar{\mathbf{y}}_N,  \bar{\mathbf{y}}-\bar{\mathbf{y}}_N\rangle]$ and write the following:
$$
\mathbb{E}_A[ \langle \bar{\mathbf{y}}-\bar{\mathbf{y}}_N,  \bar{\mathbf{y}}-\bar{\mathbf{y}}_N\rangle]
=
 \mathbb{E}_A[ \langle \frac{1}{n}\sum_{i=1}^N Z_i\mathbf{y}_i-\bar{\mathbf{y}}_N,  \frac{1}{n}\sum_{i=1}^N Z_i\mathbf{y}_i-\bar{\mathbf{y}}_N\rangle]
$$
 [/INST] Let $\mathbf{Y} = \begin{pmatrix} \mathbf{y}_1 & \cdots & \mathbf{y}_N \end{pmatrix} \in \mathbb{R}^{n \times N}$. Then notice that
\begin{align*}
\overline{\mathbf{y}} - \overline{\mathbf{y}}_N = \mathbf{Y}\left(n^{-1} \mathbf{1}_A - N^{-1}\mathbf{1}_N\right)
\end{align*}
where $\mathbf{1}_N, \mathbf{1}_A \in \mathbb{R}^{N \times 1}$ with $\mathbf{1}_N$ full of 1's, and $(\mathbf{1}_A)_i = 1$ if $i \in A$, and 0 otherwise. The only thing random in this expression is $\mathbf{1}_A$. So we have
\begin{align*}
\|\overline{\mathbf{y}} - \overline{\mathbf{y}}_N\|^2 &= \left(N^{-1}\mathbf{1}_N - n^{-1} \mathbf{1}_A\right)^\intercal \mathbf{Y}^\intercal\mathbf{Y}\left(N^{-1}\mathbf{1}_N - n^{-1} \mathbf{1}_A\right) \\
&= N^{-2}\mathbf{1}_N^\intercal \mathbf{Y}^\intercal\mathbf{Y}\mathbf{1}_N - 2(nN)^{-1}\mathbf{1}_N^\intercal\mathbf{Y}^\intercal\mathbf{Y}\mathbf{1}_A + n^{-2}\mathbf{1}_A^\intercal \mathbf{Y}^\intercal\mathbf{Y}\mathbf{1}_A \\
\end{align*}
Using the fact that $\mathbb{E}[\mathbf{1}_A] = \frac{n}{N}\mathbf{1}_N$ and $\text{Cov}(\mathbf{1}_A) = \frac{n}{N}(1 - \frac{n-1}{N-1})\mathbf{I} + \frac{n}{N}(\frac{n-1}{N-1} - \frac{n}{N})\mathbf{1}_N\mathbf{1}_N^\intercal \overset{\text{def}}{=}\Sigma$ and facts about quadratic forms, we have
\begin{align*}
\mathbb{E}\|\overline{\mathbf{y}} - \overline{\mathbf{y}}_N\|^2 &= N^{-2}\mathbf{1}_N^\intercal \mathbf{Y}^\intercal\mathbf{Y}\mathbf{1}_N - 2(nN)^{-1}\mathbf{1}_N^\intercal\mathbf{Y}^\intercal\mathbf{Y}\left(\frac{n}{N}\mathbf{1}_N\right) \\
&\qquad + n^{-2}\left(\left(\frac{n}{N}\mathbf{1}_N\right)^\intercal \mathbf{Y}^\intercal\mathbf{Y}\left(\frac{n}{N}\mathbf{1}_N\right) + \text{tr}(\mathbf{Y}^\intercal \mathbf{Y}\Sigma)\right)\\
&= n^{-2}\text{tr}(\mathbf{Y}^\intercal \mathbf{Y}\Sigma) \\
&=n^{-2}\left(\frac{n}{N}\left(1 - \frac{n-1}{N-1}\right)\text{tr}(\mathbf{Y}^\intercal\mathbf{Y}) - \frac{n}{N}\left(\frac{n}{N} - \frac{n-1}{N-1}\right)\text{tr}(\mathbf{1}_N^\intercal \mathbf{Y}^\intercal \mathbf{Y} \mathbf{1}_N)\right) \\
&= n^{-2}\left(\frac{n}{N}\left(1 - \frac{n-1}{N-1}\right)\sum_{i=1}^{N}\|\mathbf{y}_i\|^2 - \frac{n}{N}\left(\frac{n}{N} - \frac{n-1}{N-1}\right)N^2 \overline{\mathbf{y}}_N^\intercal\overline{\mathbf{y}}_N\right) \\
&= n^{-2}\left(\frac{n}{N}\left(1 - \frac{n-1}{N-1}\right)\sum_{i=1}^{N}\|\mathbf{y}_i\|^2 - \frac{n}{N}\left(1 - \frac{n-1}{N-1}\right)N \overline{\mathbf{y}}_N^\intercal\overline{\mathbf{y}}_N\right) \\
&= n^{-2}\frac{n}{N}\left(1 - \frac{n-1}{N-1}\right)\sum_{i=1}^{N}\|\mathbf{y}_i - \overline{\mathbf{y}}_N\|^2 \\
&= \frac{N-n}{n}\frac{1}{N}e
\end{align*}
where
\begin{align*}
\sum_{i=1}^{N} \|\mathbf{y}_i - \overline{\mathbf{y}}_N\|^2 &= \sum_{i=1}^{N} (\mathbf{y}_i^\intercal \mathbf{y}_i - 2 \mathbf{y}_i^\intercal \overline{\mathbf{y}}_N + \overline{\mathbf{y}}_N^\intercal\overline{\mathbf{y}}_N) \\
&= \left(\sum_{i=1}^{N} \mathbf{y}_i^\intercal \mathbf{y}_i\right) - 2 \left(\sum_{i=1}^{N}\mathbf{y}_i\right)^\intercal \overline{\mathbf{y}}_N + N \overline{\mathbf{y}}_N^\intercal\overline{\mathbf{y}}_N) \\
&= \left(\sum_{i=1}^{N} \mathbf{y}_i^\intercal \mathbf{y}_i\right) - 2 N \overline{\mathbf{y}}_N^\intercal \overline{\mathbf{y}}_N + N \overline{\mathbf{y}}_N^\intercal\overline{\mathbf{y}}_N) \\
&= \left(\sum_{i=1}^{N} \mathbf{y}_i^\intercal \mathbf{y}_i\right) - N \overline{\mathbf{y}}_N^\intercal\overline{\mathbf{y}}_N)
\end{align*}
</s>