noise injections. We obtain the activations ${{{\boldsymbol{\mathbf{}}}}{h}} = \{{{{\boldsymbol{\mathbf{}}}}{h}}_0, ... , {{{\boldsymbol{\mathbf{}}}}{h}}_{L-1} \}$, where ${{\boldsymbol{\mathbf{}}}}{h}_{0}={{\boldsymbol{\mathbf{}}}}{x}$ is the input data. Let ${{\boldsymbol{\mathbf{}}}}{\epsilon}$ be the set of noise injections at each layer: ${{\boldsymbol{\mathbf{}}}}{\epsilon} = \{{{{\boldsymbol{\mathbf{}}}}{\epsilon}}_0, ... , {{{\boldsymbol{\mathbf{}}}}{\epsilon}}_{L-1} \}$. When performing a noise injection procedure, the value of the next layer’s activations depends on the noised value of the previous layer. We denote the intermediate, soon-to-be-noised value of an activation, for a network consisting of dense layers (a.k.a. a multi-layer perceptron: MLP) as $\hat{{{\boldsymbol{\mathbf{}}}}{h}}_{i}$ and the subsequently noised value as $\tilde{{{\boldsymbol{\mathbf{}}}}{h}}_{i}$: $$\begin{aligned}
\hat{{{\boldsymbol{\mathbf{}}}}{h}}_{i}= \phi\left(\tilde{{{\boldsymbol{\mathbf{}}}}{h}}_{i-1}{{\boldsymbol{\mathbf{}}}}{W}_{i}\right)\,, \qquad  
\tilde{{{\boldsymbol{\mathbf{}}}}{h}}_{i} = \hat{{{\boldsymbol{\mathbf{}}}}{h}}_{i} \circ {{\boldsymbol{\mathbf{}}}}{\epsilon}_{i}\,, 
\label{eq:noise_recursion}\end{aligned}$$ where $\phi$ is some non-linearity, and $\circ$ is some element-wise operation. We can, for example, add or multiply Gaussian noise to each hidden layer unit. The multiplicative case can be rewritten as an activation-scaled addition[^1]. Both these cases give us: $$\begin{aligned}
\tilde{{{\boldsymbol{\mathbf{}}}}{h}}_i &= \hat{{{\boldsymbol{\mathbf{}}}}{h}}_i + {{\boldsymbol{\mathbf{}}}}{\epsilon}_i, 
\qquad {{\boldsymbol{\mathbf{}}}}{\epsilon}_i \sim \mathcal{N}(0,{{\boldsymbol{\mathbf{}}}}{\sigma}_i^2), \label{eq:noise_add} \\
\tilde{{{\boldsymbol{\mathbf{}}}}{h}}_i &= \hat{{{\boldsymbol{\mathbf{}}}}{h}}_i + {{\boldsymbol{\mathbf{}}}}{\epsilon}_i, 
\qquad {{\boldsymbol{\mathbf{}}}}{\epsilon}_i \sim \mathcal{N}\left(0,\hat{{{\boldsymbol{\mathbf{}}}}{h}}_i^2{{\boldsymbol{\mathbf{}}}}{\sigma}_i^2\right).
\label{eq:noise_mult}\end{aligned}$$

Here we focus our analysis on noise *additions*, but through equation we can translate our results to the multiplicative case. The expected cost function to be minimized by SGD is estimated by Jensen’s inequality such that: $$\begin{aligned}
\tilde{\mathcal{L}}_{\mathrm{SGD}}(\mathcal{B}; {{\boldsymbol{\mathbf{}}}}{\theta}) 