matter by \[ass:symAct\].

#### Asymptotic notations.

The expressions $f = O(g) \iff g = \Omega(f)$ have their typical meanings, and $f = \Theta(g)$ iff $f = O(g), g = O(f)$. We take $f(x) = \tilde O(g(x)) \iff g(x) = \tilde \Omega(f(x))$ to mean $f(x) = O(g\log^k x)$ for some $k \in \Z$ (this is slightly different from the standard usage of $\tilde O$), and $f = \tilde\Theta(g) \iff f = \tilde O(g) \And g = \tilde O(f).$ We introduce a new notation: $f = {{\check\Theta}}(g)$ if $f(x) = O(g(x) \cdot x^{\epsilon})$ and $f(x) = \Omega(g(x) \cdot x^{-{\epsilon}})$, as $x \to \infty$, for any ${\epsilon}> 0$. All asymptotic notations are sign-less, i.e. can indicate either positive or negative quantities, unless stated otherwise.

Overview
========

The primary reason we may say anything about the average behavior of any of the above quantities is the central limit theorem: every time the activations of the previous layer pass through an affine layer whose weights are sampled i.i.d., the output is a sum of a large number of random variables, and thus follows approximately Gaussian distributions. The mean and variance of these distributions can be computed by keeping track of the mean and variances of 