can express the effect of the Gaussian noise injection on the cost function as: $$\tilde{\mathcal{L}}_{\mathrm{SGD}}(\mathcal{B};{{\boldsymbol{\mathbf{}}}}{\theta}, {{\boldsymbol{\mathbf{}}}}{\epsilon}) =  \mathcal{L}_{\mathrm{SGD}}(\mathcal{B}; {{\boldsymbol{\mathbf{}}}}{\theta}) + \Delta\mathcal{L}({{\boldsymbol{\mathbf{}}}}{\mathcal{E}}_{L})$$ where ${{\boldsymbol{\mathbf{}}}}{\mathcal{E}}_{L}$ is the noise accumulated on the final layer $L$ from the noise additions ${{\boldsymbol{\mathbf{}}}}{\epsilon}$ on the previous hidden layer activations. We formulate an estimate for $\Delta \mathcal{L}$ that is architecture independent and is positive semi-definite (PSD), and thus a viable regulariser. It is important that the regularisation term we add to our loss is PSD. Non-PSD regularisers can change sign from batch to batch and do not give a consistent objective to optimise, making them unfit as regularisers [@Botev2017; @Sagun2018; @Wei2020].

Here we consider the case where we noise all layers except the *final* predictive layer which we consider to have no activation function for the purposes of our analysis. First and foremost we need an analytic form of the accumulated noise ${{\boldsymbol{\mathbf{}}}}{\mathcal{E}}_{L}$. We denote each layer’s Jacobian as ${{\boldsymbol{\mathbf{}}}}{J}_{k}({{\boldsymbol{\mathbf{}}}}{x}) \in \mathbb{R}^{N_L \times N_k}, \quad {{\boldsymbol{\mathbf{}}}}{J}_{k}({{\boldsymbol{\mathbf{}}}}{x})_{i,j} = \partial {{\boldsymbol{\mathbf{}}}}{h}_{L}({{\boldsymbol{\mathbf{}}}}{x})_i / \partial {{\boldsymbol{\mathbf{}}}}{h}_{k}({{\boldsymbol{\mathbf{}}}}{x})_j$ and have:

Consider an $L$ layer neural network, experiencing GNIs ${{\boldsymbol{\mathbf{}}}}{\epsilon}_{i}$ at each layer $i \in [0, \dots, L - 1]$. Assuming the Hessians, of the form $\nabla^2 {{\boldsymbol{\mathbf{}}}}{h}_{l}({{\boldsymbol{\mathbf{}}}}{x})\rvert_{{{\boldsymbol{\mathbf{}}}}{h}_{m}({{\boldsymbol{\mathbf{}}}}{x})}, m < l$ where $l,m$ index 