\ra = \la (\pd E/ \pd \p x l _j)^2 \ra$ for any $i, j, l$.

One can see that \[ass:symAct\](a) is satisfied if the input $\p x 0 \in \{\pm 1\}^N$ and \[ass:symAct\](b) is satisfied if \[ass:gradInd\] below is true and the gradient at the last layer $\pd E/\pd x L \in \{\pm 1 \}^N$. But in general it is justified both empirically and theoretically as an approximation, because $(\p h l _i)^2 - (\p h l _j)^2$ stays about constant with $l$, but $(\p h l _i)^2$ and $(\p h l _j)^2$ grow rather quickly at the same pace with $l$ (as will be seen later in calculations), so that their additive difference becomes negligible; similarly for $(\p x l _i)^2$ and $(\pd E/\pd \p h l _i)^2$.

\[ass:gradInd\] (a) We assume the we use a different set of weights for backpropagation than those used to compute the network outputs, but sampled i.i.d. from the same distributions. (b) For any loss function $E$, we assume that the gradient at layer $l$, $\pd E/\pd \p x l _i$, is independent from all activations $\p h {l} _j$ and $\p x {l-1} _j$ from the previous layer.

\[ass:gradInd\](a) was first 