<p>You could try if <a href="https://arxiv.org/abs/1502.03167" rel="nofollow noreferrer">this paper</a> helps. 
They say that if you don't use normalization, you need to train "more carefully", meaning using a lower learning rate. </p>

<p>Skimming the first pages, I could imagine it works like this: </p>

<p>For some nonlinearities, there's a 'good input value range', and batch norm brings values into that range. High input values are bad and lead to saturation (little slope in the function and "vanishing gradients").</p>

<p>So, if you don't normalize, you need to make smaller steps - a lower learning rate - to avoid 'jumping' into weights that lead to high values within the net. And also be more careful how you initialize weights. I guess if you use ReLus, that's not so much of a problem. But please correct me if someone else has had different experiences with ReLus.</p>
caffe batchnorm layer affect base_lr?
<p>I saw the following code under caffe framework. The whole code is trying to write caffe <code>train_val.prototxt</code> and <code>solver.prototxt</code>.</p>

<pre><code># Use different initial learning rate.
if use_batchnorm:
    base_lr = 0.0004
else:
    base_lr = 0.00004
</code></pre>

<p>Why is the base learning rate different?</p>