\geq \mathbb{E}_{\substack{\tilde{{{\boldsymbol{\mathbf{}}}}{h}} \sim p(\tilde{{{\boldsymbol{\mathbf{}}}}{h}}|{{\boldsymbol{\mathbf{}}}}{h}) \\
{{\boldsymbol{\mathbf{}}}}{x},{{\boldsymbol{\mathbf{}}}}{y} \sim \mathcal{B}}} \log   p_{{{\boldsymbol{\mathbf{}}}}{\theta}}({{\boldsymbol{\mathbf{}}}}{y}|\tilde{{{\boldsymbol{\mathbf{}}}}{h}}) 
\label{eq:invariant_form_jensen}\end{aligned}$$ with $p(\tilde{{{\boldsymbol{\mathbf{}}}}{h}}|\hat{{{\boldsymbol{\mathbf{}}}}{h}})=\prod_i^L p(\tilde{{{\boldsymbol{\mathbf{}}}}{h}}_i|\hat{{{\boldsymbol{\mathbf{}}}}{h}}_i, {{\boldsymbol{\mathbf{}}}}{\epsilon}_i)$ and $p(\tilde{{{\boldsymbol{\mathbf{}}}}{h}}_i|\hat{{{\boldsymbol{\mathbf{}}}}{h}}_i, {{\boldsymbol{\mathbf{}}}}{\epsilon}_i)=  \mathcal{N}(\tilde{{{\boldsymbol{\mathbf{}}}}{h}_i}|\hat{{{\boldsymbol{\mathbf{}}}}{h}}_i,{{\boldsymbol{\mathbf{}}}}{\sigma}^2)$.

We restrict our analysis to networks with non-linearities that are broadly moment preserving [@Vladimirova2019UnderstandingLevel], such that the accumulated noise at a given layer is still Gaussian. The properties of the accumulated noise on $\tilde{{{\boldsymbol{\mathbf{}}}}{h}}_n$ depends on the properties of the activation function $\phi(\cdot)$ used in the network. For instance, Gaussian noise becomes sub-Gaussian when fed through a bounded activation function, such as a sigmoid, by Hoeffding’s Lemma [@Bentkus2004; @Vladimirova2019UnderstandingLevel]. A class of functions that are broadly moment preserving are those that obey the *extended envelope property*, as they have *asymptotically equivalent moments*, see Appendix A and [@Vladimirova2019UnderstandingLevel]. Activation functions that satisfy this property are among the most widely used in deep learning (Rectified Linear Units ($\mathrm{ReLU}$) and Exponential Linear Units ($\mathrm{ELU}$) among others). Even more complex operations, such as max-pooling in convolutional networks, satisfy this property [@Vladimirova2019UnderstandingLevel].

### Sensitivity and Classification Margins {#sec:sensitivity_background}

Sophisticated deep learning models can be brittle [@Szegedy2014; @Shamir2019; @Goodfellow2015; @Papernot2016; @Moosavi-Dezfooli2016]. Adding small unstructured noise to data, perturbations that would not fool a human can alter neural network predictions. A model’s weakness to perturbations is called the 