<p>Pretty much every network uses batch normalization, which is exactly that. Paper can be found here: (<a href="https://arxiv.org/abs/1502.03167" rel="nofollow noreferrer">https://arxiv.org/abs/1502.03167</a>). In essence it normalizes the values to be 0 mean and unit variance before being fed into the next layer. Another work is on self normalizing linear units (selu), which in some sense does this automatically without needing any kind of scaling. Paper can be found here: (<a href="https://arxiv.org/abs/1706.02515" rel="nofollow noreferrer">https://arxiv.org/abs/1706.02515</a>). </p>
Interlayer scaling or normalisation between hidden layers in ANNs CNNs and MLPs
<p>Would anyone here know if there is any kind of normalisation or scaling between layers in existing Neural Network arcitectures?</p>

<p>Scaling inputs is common and i am familiar with ReLU blow up. Most models i see indicate a small range of values like -2 to +2 but i don't see how this can be maintained from layer to layer. Irrespective of the activation function the second layer output is in the tens then the third layer is hundreds and final output is tens of thousands. In the worst case the layer returns NaN. A work around can be by scaling or alternating ReLU/sigmoid but I would like to know if this is this common?</p>