<p>What the <a href="https://arxiv.org/abs/1502.03167" rel="nofollow noreferrer">original paper</a> tries to explain is to reduce overfitting use Batch Normalization.</p>

<blockquote>
  <p>Where should you splice the normalization when designing a network?</p>
</blockquote>

<p>Set the normalization early on inputs. Unbalanced input extreme values can cause instability. </p>

<p>While if you normalize on outputs this will not prevent the inputs to cause the instability all over again.</p>

<p>Here is the little code that explains what the BN do:</p>

<pre><code>import torch
import torch.nn as nn

m = nn.BatchNorm1d(100, affine=False)
input = 1000*torch.randn(3, 100)
print(input)
output = m(input)
print(output)
print(output.mean()) # should be ~ 0
print(output.std()) # should be ~ 1
</code></pre>

<blockquote>
  <p>Does it make sense to normalize any time after you have a dense layer</p>
</blockquote>

<p>Yes, you may do so as matrix multiplication may lead to producing the extremes. Also, after convolution layers, because these are also matrix multiplication, similar but less intense comparing to dense (<code>nn.Linear</code>) layer. If you for instance print the resent model, you will see that batch norms are set every time after the conv layer like this:</p>

<pre><code>(conv1): Conv2d(3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)
(bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
</code></pre>

<p>To print the full resnet you may use this:</p>

<pre><code>import torchvision.models as models
r = models.resnet18()
print(r)
</code></pre>
When to use layernorm/batch norm?
<p>Where should you splice the normalization when designing a network? E.g. if you have a stacked Transformer or Attention network, does it make sense to normalize any time after you have a dense layer?</p>