process for each layer, and assuming that all Hessians of the form $\nabla^2 {{\boldsymbol{\mathbf{}}}}{h}_l\rvert_{{{\boldsymbol{\mathbf{}}}}{h}_m({{\boldsymbol{\mathbf{}}}}{x})} \quad \forall m < l$ are finite, we obtain for a layer $k$: $${{\boldsymbol{\mathbf{}}}}{\mathcal{E}}_{k} = \sum_{i=0}^{k-1}\frac{\partial {{\boldsymbol{\mathbf{}}}}{h}_k}{\partial {{\boldsymbol{\mathbf{}}}}{h}_{i}}{{\boldsymbol{\mathbf{}}}}{\epsilon}_{i} + \mathcal{O}({{\boldsymbol{\mathbf{}}}}{\kappa_k})
\label{eqapp:cumulative_noise}$$

where once again, $\mathcal{O}({{\boldsymbol{\mathbf{}}}}\kappa_k)$ represents asymptotically dominated higher order terms as the variance of all ${{\boldsymbol{\mathbf{}}}}{\epsilon}_k, k\in[0,\dots,k]$ tends to 0.

Let ${{\boldsymbol{\mathbf{}}}}{J}_{k}({{\boldsymbol{\mathbf{}}}}{x}) \in \mathbb{R}^{N_L \times N_k}$ ($N_L$ the number of output neurons and $N_k$ the number of neurons in layer $k$) be defined element-wise as $({{\boldsymbol{\mathbf{}}}}{J}_{k}({{\boldsymbol{\mathbf{}}}}{x}))_{i,j} = \frac{\partial ({{\boldsymbol{\mathbf{}}}}{h}_{L}({{\boldsymbol{\mathbf{}}}}{x}))_i}{\partial ({{\boldsymbol{\mathbf{}}}}{h}_{k}({{\boldsymbol{\mathbf{}}}}{x}))_j}$. The accumulated noise at layer $L$ is: $${{\boldsymbol{\mathbf{}}}}{\mathcal{E}}_{L} = \sum_{k=0}^{L-1}{{\boldsymbol{\mathbf{}}}}{J}_{k}({{\boldsymbol{\mathbf{}}}}{x}){{\boldsymbol{\mathbf{}}}}{\epsilon}_{k} + \mathcal{O}({{\boldsymbol{\mathbf{}}}}{\kappa}_L)
\label{eqapp:cumulative_noise_app}$$

For simplicity of notation we denote $\mathcal{O}({{\boldsymbol{\mathbf{}}}}{\kappa}_L)$ as $\mathcal{O}({{\boldsymbol{\mathbf{}}}}{\gamma})$ completing the proof.

Explicit Regularisation Derivation {#app:exp_reg}
==================================

Consider an $L$ layer neural network experiencing GNIs ${{\boldsymbol{\mathbf{}}}}{\epsilon}_{i}$ at each layer $i \in [0, \dots, L - 1]$. We assume the the third derivative of $\mathcal{L}({{\boldsymbol{\mathbf{}}}}{x})$ w.r.t ${{\boldsymbol{\mathbf{}}}}{h}_{L}({{\boldsymbol{\mathbf{}}}}{x})$ is finite, which is the case for most loss functions. Using the form of ${{\boldsymbol{\mathbf{}}}}{\mathcal{E}}_{L}$ in Proposition \[appprop:accum\_noise\]; we can marginalise out the injected noise ${{\boldsymbol{\mathbf{}}}}{\epsilon}$ to obtain an added regulariser: $$\mathbb{E}_{{{\boldsymbol{\mathbf{}}}}{\epsilon}} \left[ \Delta \mathcal{L}({{\boldsymbol{\mathbf{}}}}{\mathcal{E}}_{L}) \right] = \frac{1}{2}\mathbb{E}_{{{\boldsymbol{\mathbf{}}}}{x} \sim \mathcal{B}} \left[ \sum_{k=0}^{L-1}\sigma^2_k \mathrm{Tr}({{\boldsymbol{\mathbf{}}}}{J}^T_{k}({{\boldsymbol{\mathbf{}}}}{x})
    {{\boldsymbol{\mathbf{}}}}{H}_{L}({{\boldsymbol{\mathbf{}}}}{x}){{\boldsymbol{\mathbf{}}}}{J}_{k}({{\boldsymbol{\mathbf{}}}}{x}))  \right] +   O({{\boldsymbol{\mathbf{}}}}{\kappa}) 
    \label{eqn:deltaL}$$ where $\mathcal{L}({{\boldsymbol{\mathbf{}}}}{x})$ is the loss for a datapoint 