\forall k \geq 1
\label{eqapp:moment-preservation}$$

where $\|\cdot\|_k$ is the $k^{th}$ norm.

Accumulated Noise Derivation {#app:accum_noise}
============================

For an $L$ layer neural network, experiencing GNIs ${{\boldsymbol{\mathbf{}}}}{\epsilon}_{i}$ at each layer $i \in [0, \dots, L - 1]$, the the noise accumulated at the final layer, ${{\boldsymbol{\mathbf{}}}}{\mathcal{E}}_{L}$, is given by: $${{\boldsymbol{\mathbf{}}}}{\mathcal{E}}_{L} = \sum_{k=0}^{L-1}{{\boldsymbol{\mathbf{}}}}{\epsilon}_{k}{{\boldsymbol{\mathbf{}}}}{J}_{k}({{\boldsymbol{\mathbf{}}}}{x}) + \mathcal{O} ({{\boldsymbol{\mathbf{}}}}\kappa_L)$$ $\mathcal{O} ({{\boldsymbol{\mathbf{}}}}\kappa_L)$ represents higher order terms that tend to zero in the limit of small variance noise injections. \[appprop:accum\_noise\]

Recall that ${{\boldsymbol{\mathbf{}}}}{h}$ denotes the vanilla activations of the network, those we obtain with no noise injection. If we consider layer 0 to be the data ${{\boldsymbol{\mathbf{}}}}{x}$, we can define the accumulated noise on a layer by recursion. At layer 0 we have we have no accumulated noise: ${{\boldsymbol{\mathbf{}}}}{\mathcal{E}}_{0}=0$. Here we can invoke Taylor’s theorem on ${{\boldsymbol{\mathbf{}}}}{h}_1({{\boldsymbol{\mathbf{}}}}{x} + {{\boldsymbol{\mathbf{}}}}{\epsilon}_0)$ around the original input ${{\boldsymbol{\mathbf{}}}}x$. Namely, if we assume that all terms in Hessian of ${{\boldsymbol{\mathbf{}}}}{h}_1({{\boldsymbol{\mathbf{}}}}{x})$ are finite (i.e. $|\partial^2 {{\boldsymbol{\mathbf{}}}}{h}_1({{\boldsymbol{\mathbf{}}}}x)_i / \partial {{\boldsymbol{\mathbf{}}}}x_j {{\boldsymbol{\mathbf{}}}}x_k|<\infty\, \forall i,j,k$)

$${{\boldsymbol{\mathbf{}}}}{h}_1({{\boldsymbol{\mathbf{}}}}{x} + {{\boldsymbol{\mathbf{}}}}{\epsilon}_0) = {{\boldsymbol{\mathbf{}}}}{h}_1({{\boldsymbol{\mathbf{}}}}{x}) + \frac{\partial {{\boldsymbol{\mathbf{}}}}{h}_1}{\partial {{\boldsymbol{\mathbf{}}}}{x}}{{\boldsymbol{\mathbf{}}}}{\epsilon}_0 + \mathcal{O}({{\boldsymbol{\mathbf{}}}}{\kappa}_0)
\label{eqapp:layer_taylor_expansion}$$

where once again, $\mathcal{O}({{\boldsymbol{\mathbf{}}}}\kappa_0)$ represents asymptotically dominated higher order terms as the variance of ${{\boldsymbol{\mathbf{}}}}{\epsilon}_0$ tends to 0. As such we have ${{\boldsymbol{\mathbf{}}}}{\mathcal{E}}_{1}= \frac{\partial {{\boldsymbol{\mathbf{}}}}{h}_1}{\partial {{\boldsymbol{\mathbf{}}}}{x}}{{\boldsymbol{\mathbf{}}}}{\epsilon}_0 + \mathcal{O}({{\boldsymbol{\mathbf{}}}}{\kappa}_0)$.

Repeating this 