${{\boldsymbol{\mathbf{}}}}{x}$, ${{\boldsymbol{\mathbf{}}}}{H}_{L}({{\boldsymbol{\mathbf{}}}}{x})$ is $\nabla^2 \mathcal{L}\rvert_{{{\boldsymbol{\mathbf{}}}}{h}_{L}({{\boldsymbol{\mathbf{}}}}{x})} \in \mathbb{R}^{N_L
\times N_L}$ ($N_L$ is the number of output neurons), and $O({{\boldsymbol{\mathbf{}}}}{\kappa})$ represents higher order terms in ${{\boldsymbol{\mathbf{}}}}{\mathcal{E}}_{L}$ that disappear in the limit of small variance noise. \[appprop:explicit\_reg\]

Recall that ${{\boldsymbol{\mathbf{}}}}{h}$ denotes the vanilla activations of the network, those we obtain with no noise injection. Let us *not* inject noise in the final, predictive, layer of our network such that the noise on this layer is accumulated from the noising of previous layers.

Our network loss can be defined in terms of our final layer activation ${{\boldsymbol{\mathbf{}}}}{h}_{L}({{\boldsymbol{\mathbf{}}}}{x})$ for a datapoint ${{\boldsymbol{\mathbf{}}}}{x}$, using Taylor’s Theorem. Denoting $\nabla^2 \mathcal{L}\rvert_{{{\boldsymbol{\mathbf{}}}}{h}_{L}({{\boldsymbol{\mathbf{}}}}{x})} \in \mathbb{R}^{N_L
\times N_L}$ ($N_L$ is the number of output neurons) as ${{\boldsymbol{\mathbf{}}}}{H}_{L}({{\boldsymbol{\mathbf{}}}}{x})$ we have, and assuming that third derivatives of the loss w.r.t ${{\boldsymbol{\mathbf{}}}}{h}_L$ are finite ($|\partial^3 \mathcal{L}({{\boldsymbol{\mathbf{}}}}x) / \partial {{\boldsymbol{\mathbf{}}}}{h}_L({{\boldsymbol{\mathbf{}}}}x)_i {{\boldsymbol{\mathbf{}}}}{h}_L({{\boldsymbol{\mathbf{}}}}x)_j {{\boldsymbol{\mathbf{}}}}{h}_L({{\boldsymbol{\mathbf{}}}}x)_k|<\infty\, \forall i,j,k$) $$\begin{aligned}
    \mathcal{L}({{\boldsymbol{\mathbf{}}}}{h}_{L}({{\boldsymbol{\mathbf{}}}}{x}) + {{\boldsymbol{\mathbf{}}}}{\mathcal{E}}_{L}({{\boldsymbol{\mathbf{}}}}{x})) &=  \mathcal{L}({{\boldsymbol{\mathbf{}}}}{h}_{L}({{\boldsymbol{\mathbf{}}}}{x})) +  \nabla \mathcal{L}\rvert_{{{\boldsymbol{\mathbf{}}}}{h}_{L}({{\boldsymbol{\mathbf{}}}}{x})}{{\boldsymbol{\mathbf{}}}}{\mathcal{E}}_{L}({{\boldsymbol{\mathbf{}}}}{x}) + \frac{1}{2}{{\boldsymbol{\mathbf{}}}}{\mathcal{E}}^T_{L}({{\boldsymbol{\mathbf{}}}}{x}) {{\boldsymbol{\mathbf{}}}}{H}_{L}({{\boldsymbol{\mathbf{}}}}{x}) {{\boldsymbol{\mathbf{}}}}{\mathcal{E}}_{L}({{\boldsymbol{\mathbf{}}}}{x}) + \mathcal{O}({{\boldsymbol{\mathbf{}}}}{\kappa})\nonumber \\
    &=  \mathcal{L}({{\boldsymbol{\mathbf{}}}}{x}) + \nabla \mathcal{L}\rvert_{{{\boldsymbol{\mathbf{}}}}{h}_{L}({{\boldsymbol{\mathbf{}}}}{x})}{{\boldsymbol{\mathbf{}}}}{\mathcal{E}}_{L}({{\boldsymbol{\mathbf{}}}}{x}) + \frac{1}{2}{{\boldsymbol{\mathbf{}}}}{\mathcal{E}}^T_{L}({{\boldsymbol{\mathbf{}}}}{x}) {{\boldsymbol{\mathbf{}}}}{H}_{L}({{\boldsymbol{\mathbf{}}}}{x}) {{\boldsymbol{\mathbf{}}}}{\mathcal{E}}_{L}({{\boldsymbol{\mathbf{}}}}{x}) + \mathcal{O}({{\boldsymbol{\mathbf{}}}}{\kappa})
    \label{eqapp:2nd_order_taylor}\end{aligned}$$ Where $\mathcal{O}({{\boldsymbol{\mathbf{}}}}\kappa)$ represents asymptotically dominated higher order terms that go to zero in the limit of small variance noise injections. The second equality comes from the fact that the loss from the vanilla 