penalisation of the norms of the Jacobians, induced by GNIs for each layer $k$, as an unpweighted penalisation of $\|{{\boldsymbol{\mathbf{}}}}{J}_0({{\boldsymbol{\mathbf{}}}}{x})\|_F$. By the chain rule ${{\boldsymbol{\mathbf{}}}}{J}_0$ can be expressed in terms of any of the other network Jacobians ${{\boldsymbol{\mathbf{}}}}{J}_0({{\boldsymbol{\mathbf{}}}}{x}) = {{\boldsymbol{\mathbf{}}}}{J}_k({{\boldsymbol{\mathbf{}}}}{x})\frac{\partial {{\boldsymbol{\mathbf{}}}}{h}_k}{{{\boldsymbol{\mathbf{}}}}{x}} \forall k \in [0 \dots L]$. We can write $\|{{\boldsymbol{\mathbf{}}}}{J}_0({{\boldsymbol{\mathbf{}}}}{x})\|_F = \|{{\boldsymbol{\mathbf{}}}}{J}_k({{\boldsymbol{\mathbf{}}}}{x})\frac{\partial {{\boldsymbol{\mathbf{}}}}{h}_k}{{{\boldsymbol{\mathbf{}}}}{x}}\|_F \leq \|{{\boldsymbol{\mathbf{}}}}{J}_k({{\boldsymbol{\mathbf{}}}}{x})\|_F\|\frac{\partial {{\boldsymbol{\mathbf{}}}}{h}_k}{{{\boldsymbol{\mathbf{}}}}{x}}\|_F$. Minimising $\|{{\boldsymbol{\mathbf{}}}}{J}_0({{\boldsymbol{\mathbf{}}}}{x})\|_F$ is equivalent to minimising $\|{{\boldsymbol{\mathbf{}}}}{J}_k({{\boldsymbol{\mathbf{}}}}{x})\|_F$ and $\|\frac{\partial {{\boldsymbol{\mathbf{}}}}{h}_k}{{{\boldsymbol{\mathbf{}}}}{x}}\|_F$, and upweighted penalisations of $\|{{\boldsymbol{\mathbf{}}}}{J}_k({{\boldsymbol{\mathbf{}}}}{x})\|_F$ should translate into a shrinkage of $\|{{\boldsymbol{\mathbf{}}}}{J}_0({{\boldsymbol{\mathbf{}}}}{x})\|_F$.

To illustrate this, consider the artificial situation where the Jacobians of the network are invertible. Recall that by the chain rule ${{\boldsymbol{\mathbf{}}}}{J}_i({{\boldsymbol{\mathbf{}}}}{x}){{\boldsymbol{\mathbf{}}}}{J}_0^{-1}({{\boldsymbol{\mathbf{}}}}{x}) = (\frac{\partial {{\boldsymbol{\mathbf{}}}}{h}_k}{{{\boldsymbol{\mathbf{}}}}{x}})^{-1} = \nabla {{\boldsymbol{\mathbf{}}}}{h}_k^{-1}\rvert_{{{\boldsymbol{\mathbf{}}}}{x}}$ (for any k indexing over layers). Given the form of the accumulated noise in Proposition \[prop:accum\_noise\], we can state that adding noise to each layer is broadly equivalent to adding the following noise to data: $$\epsilon_0 \sim \mathcal{N}(0,{{\boldsymbol{\mathbf{}}}}{\sigma}_0^2 +  \sum_{i=1}^{L-1}(\nabla {{\boldsymbol{\mathbf{}}}}{h}_k^{-1}\rvert_{{{\boldsymbol{\mathbf{}}}}{x}}{{\boldsymbol{\mathbf{}}}}{\sigma}^T_k)^2))
\label{eq:equiv_variance_input}$$

In this case, training with GNIs is equivalent to training with *anisotropic* noise with variance ${{\boldsymbol{\mathbf{}}}}{\Sigma}({{\boldsymbol{\mathbf{}}}}{x})^2 = {{\boldsymbol{\mathbf{}}}}{\sigma}^2(1 +  \sum_{i=1}^{L-1}(\nabla {{\boldsymbol{\mathbf{}}}}{h}_k^{-1}\rvert_{x})^2)$. For an equivalent model that only adds noise with variance ${{\boldsymbol{\mathbf{}}}}{\sigma}^2$ to data, we can state that ${{\boldsymbol{\mathbf{}}}}{\Sigma}^2({{\boldsymbol{\mathbf{}}}}{x}) \geq {{\boldsymbol{\mathbf{}}}}{\sigma}^2$, element-wise. Substituting ${{\boldsymbol{\mathbf{}}}}{\Sigma}^2({{\boldsymbol{\mathbf{}}}}{x})$ into equation , we can see that 