over layers, are finite, the noise accumulated at the final layer, ${{\boldsymbol{\mathbf{}}}}{\mathcal{E}}_{L}$, is given by: $${{\boldsymbol{\mathbf{}}}}{\mathcal{E}}_{L} = \sum_{k=0}^{L-1}{{\boldsymbol{\mathbf{}}}}{\epsilon}_{k}{{\boldsymbol{\mathbf{}}}}{J}_{k}({{\boldsymbol{\mathbf{}}}}{x}) + \mathcal{O} ({{\boldsymbol{\mathbf{}}}}\gamma)$$ $\mathcal{O} ({{\boldsymbol{\mathbf{}}}}\gamma)$ represents higher order terms in ${{\boldsymbol{\mathbf{}}}}{\epsilon}$ that tend to zero in the limit of small variance noise. \[prop:accum\_noise\]

See Appendix \[app:accum\_noise\] for a proof. Using this result we can now derive the explicit regulariser induced by GNIs by marginalising out the injected noise ${{\boldsymbol{\mathbf{}}}}{\epsilon}$ from the term $\Delta \mathcal{L}({{\boldsymbol{\mathbf{}}}}{\mathcal{E}}_{L})$.

Consider an $L$ layer neural network experiencing GNIs ${{\boldsymbol{\mathbf{}}}}{\epsilon}_{i}$ at each layer $i \in [0, \dots, L - 1]$. We assume the third derivative of $\mathcal{L}({{\boldsymbol{\mathbf{}}}}{x})$ w.r.t ${{\boldsymbol{\mathbf{}}}}{h}_{L}({{\boldsymbol{\mathbf{}}}}{x})$ is finite, which is the case for most loss functions (see proof for details). Using the form of ${{\boldsymbol{\mathbf{}}}}{\mathcal{E}}_{L}$ in Proposition \[prop:accum\_noise\]; we can marginalise out the injected noise ${{\boldsymbol{\mathbf{}}}}{\epsilon}$ to obtain an added regulariser: $$\mathbb{E}_{{{\boldsymbol{\mathbf{}}}}{\epsilon}} \left[ \Delta \mathcal{L}({{\boldsymbol{\mathbf{}}}}{\mathcal{E}}_{L}) \right] = \frac{1}{2}\mathbb{E}_{{{\boldsymbol{\mathbf{}}}}{x} \sim \mathcal{B}} \left[ \sum_{k=0}^{L-1}\sigma^2_k \mathrm{Tr}({{\boldsymbol{\mathbf{}}}}{J}^T_{k}({{\boldsymbol{\mathbf{}}}}{x})
    {{\boldsymbol{\mathbf{}}}}{H}_{L}({{\boldsymbol{\mathbf{}}}}{x}){{\boldsymbol{\mathbf{}}}}{J}_{k}({{\boldsymbol{\mathbf{}}}}{x}))  \right] +   O({{\boldsymbol{\mathbf{}}}}{\kappa}) 
    \label{eqn:deltaL}$$ where $\mathcal{L}({{\boldsymbol{\mathbf{}}}}{x})$ is the loss for a datapoint ${{\boldsymbol{\mathbf{}}}}{x}$, ${{\boldsymbol{\mathbf{}}}}{H}_{L}({{\boldsymbol{\mathbf{}}}}{x})$ is $\nabla^2 \mathcal{L}\rvert_{{{\boldsymbol{\mathbf{}}}}{h}_{L}({{\boldsymbol{\mathbf{}}}}{x})} \in \mathbb{R}^{N_L
\times N_L}$ ($N_L$ is the number of output neurons), and $O({{\boldsymbol{\mathbf{}}}}{\kappa})$ represents higher order terms in ${{\boldsymbol{\mathbf{}}}}{\mathcal{E}}_{L}$ that disappear in the limit of small variance noise. \[prop:explicit\_reg\]

See Appendix \[app:exp\_reg\] for a proof 