as $N \to \infty$ (large width limit).

Note that in our definition, the index $1$ does not matter by \[ass:symAct\].

\[defn:corr\] Fix two inputs $\p x 0$ and $\p x 0{}'$. We write $\bullet'$ to denote a quantity $\bullet$ with respect to the input ${\p x 0}'$. Then define [**the correlation quantities**]{} $\p {\boldsymbol{\oldgamma}}l:= \la \p h l_1 \p h l_1{}' \ra$ and $\p {\boldsymbol{\oldlambda}}l:= \la \p x l_1 \p x l_1{}'\ra$ for $l > 0$ and $\p {\boldsymbol{\oldgamma}}0 = \p x 0 \cdot \p x 0 {}' / N$, where the expectations $\la \bullet \ra$ are taken over all random initialization of weights and biases for all layers $l$, as $N \to \infty$ (large width limit). Again, here the index $1$ does not matter by \[ass:symAct\]. By [**metric expressivity**]{}, we mean $\p {\mathbf{s}}l := \f 1 {2N} \la \|\p x l - \p x l {}'\|^2\ra = \f 1 {2N} (\la \|\p x l\|^2\ra + \la \|\p x l {}'\|^2 \ra - 2 \la \p x l \cdot \p x l {}'\ra) = \f 1 2 (\p {\mathbf{p}}l + \p {\mathbf{p}}l {}') - \p {\boldsymbol{\oldgamma}}l$. Additionally, define [**the cosine distance quantities**]{} $\p {\mathbf{e}}l := \p {\boldsymbol{\oldgamma}}l / \sqrt{\p {\mathbf{p}}l 