We derive an analytic form for the explicit regulariser induced by GNIs.

-   We establish a connection between this explicit regulariser and Tikhonov regularisation, showing that it penalises neural networks with high-frequency content in the Fourier domain.

-   Finally, we show analytically and empirically that this regularisation induces larger classification margins and better calibration of models.

Background
==========

Gaussian Noise Injections {#sec:sgd_under_injections}
-------------------------

Training a neural network involves optimising network parameters to maximise the marginal likelihood of a set of labels ${{\boldsymbol{\mathbf{}}}}{y}$ given features ${{\boldsymbol{\mathbf{}}}}{x}$ via gradient descent. With a training dataset composed of $N$ data-label pairs $\mathcal{D} = \{({{\boldsymbol{\mathbf{}}}}{x}_0, {{\boldsymbol{\mathbf{}}}}{y}_0) ... ({{\boldsymbol{\mathbf{}}}}{x}_{N-1}, {{\boldsymbol{\mathbf{}}}}{y}_{N-1})\}$ and a feed-forward neural network with $D$ parameters divided into $L$ layers: ${{\boldsymbol{\mathbf{}}}}{\theta} = \{{{\boldsymbol{\mathbf{}}}}{W}_1,...,{{\boldsymbol{\mathbf{}}}}{W}_L\}$, ${{\boldsymbol{\mathbf{}}}}{\theta} \in \mathbb{R}^D$, our objective is to minimise the expected negative log likelihood and find the optimal set of parameters ${{\boldsymbol{\mathbf{}}}}{\theta}^*$ satisfying: $$\begin{aligned}
    {{\boldsymbol{\mathbf{}}}}{\theta}_* = \operatorname*{arg\,min}_{{{\boldsymbol{\mathbf{}}}}{w}} \mathcal{L}( \mathcal{D}; {{\boldsymbol{\mathbf{}}}}{w}),
    \qquad
    \mathcal{L}( \mathcal{D}; {{\boldsymbol{\mathbf{}}}}{w}):=- \mathbb{E}_{{{\boldsymbol{\mathbf{}}}}{x},{{\boldsymbol{\mathbf{}}}}{y} \sim \mathcal{D}}\left[ \log p_{{{\boldsymbol{\mathbf{}}}}{w}}({{\boldsymbol{\mathbf{}}}}{y}|{{\boldsymbol{\mathbf{}}}}{x})\right]\,.
    \label{eq:marginal_likelihood}\end{aligned}$$

Under stochastic optimisation algorithms, such as Stochastic Gradient Descent (SGD), we estimate equation by sampling a mini-batch of data-label pairs $\mathcal{B} \subset \mathcal{D}$. $$\mathcal{L}_{\mathrm{SGD}}(\mathcal{B}; {{\boldsymbol{\mathbf{}}}}{\theta}) = -\mathbb{E}_{{{\boldsymbol{\mathbf{}}}}{x},{{\boldsymbol{\mathbf{}}}}{y} \sim \mathcal{B}} \log p_{{{\boldsymbol{\mathbf{}}}}{\theta}}({{\boldsymbol{\mathbf{}}}}{y}|{{\boldsymbol{\mathbf{}}}}{x}) \approx \mathcal{L}( \mathcal{D}; {{\boldsymbol{\mathbf{}}}}{\theta})$$

Consider an $L$ layer network with no 