widely used regularisation method operates and paves the way for designing new regularisation methods which take advantage of our findings. Regularisation methods produce models that are not only less likely to overfit, but also have better calibrated predictions that are more robust to adversarial attack. As such improving our understanding of such methods is critical as machine learning models become increasingly ubiquitous and embedded in decision making.

Extended Envelope Property {#app:eep}
==========================

As described in @Vladimirova2019UnderstandingLevel, we make us of the Extended Envelope Property.

[Extended Envelope Property:]{} A non-linear function $\phi:\mathbb{R} \to \mathbb{R}$ is said to obey the *extended envelope property* if $\exists~c_1,c_2 \geq 0 ~ d_1, d_2 \geq 0 $ such that:

-   $|\phi(x)| \geq c_1 +d_1|x|, ~ \forall x\in \mathbb{R}^+ ~\mathrm{or} ~ x\in \mathbb{R}^-$

-   $|\phi(x)| \leq c_2 +d_2|x|, \forall x\in \mathbb{R}$

Further we also use the related idea of Asymptotic Equivalence.

[Asymptotic Equivalence]{} Two sequences $a_k$, $b_k$ are asymptotically equivalent ($a_k \asymp b_k$) if $\exists d>0, D>0$ such that: $d \leq \frac{a_k}{b_k} \leq D ~ \forall k \in \mathbb{N}$.

For any non-linearity $\phi: \mathbb{R} \to \mathbb{R} $ that obeys the extended envelope property, then for any *symmetric* random variable ${{\boldsymbol{\mathbf{}}}}{X}$ we can write:

$$\|\phi({{\boldsymbol{\mathbf{}}}}{X})\|_k \asymp \|{{\boldsymbol{\mathbf{}}}}{X}\|_k ~ 