<p>There is no problem by combining Batch Normalization with ReLUs, this is done very often with no problems. For example the first paper about <a href="https://arxiv.org/abs/1512.03385" rel="nofollow noreferrer">Residual Networks</a> does this and obtains very good results in ImageNet classification.</p>
Neural networks: Combining ReLus and Batch normalization?
<p>In <a href="https://datascience.stackexchange.com/questions/14352/how-are-deep-learning-nns-different-now-2016-from-the-ones-i-studied-just-4-ye">https://datascience.stackexchange.com/questions/14352/how-are-deep-learning-nns-different-now-2016-from-the-ones-i-studied-just-4-ye</a> I was told that one should use Batch normalization:</p>

<blockquote>
  <p>It's been known for a while that NNs train best on data that is normalized --- i.e., there is zero mean and unit variance.</p>
</blockquote>

<p>I was also told one should use ReLu neurons - especially if the inputs are images. Images usually have numbers between 0 and 1 or 0 and 255.</p>

<p><strong>Question:</strong> Is it wise to combine ReLus with Batch normalizsation?</p>

<p>I would imagine that if I do first Batch normalization, I fear, one might loose information once it passes the ReLus.of have of my information once</p>