<p>In CNN for images, normalization within channel is helpful because weights are shared across channels.
The figure from another paper shows how we are dealing with BN. It's helpful to understand better.</p>
<p><img src="https://i.stack.imgur.com/DLwRc.png" alt="figure from Group Normalization paper" /></p>
<p>Figure taken from</p>
<blockquote>
<p>Wu, Y. and He, K., 2018. Group normalization. arXiv preprint arXiv: 1803.08494.</p>
</blockquote>
Why batch normalization over channels only in CNN
<p>I am wondering, if in Convolutional Neural Networks batch normalization should be applied with respect to every pixel separately, or should I take the mean of pixels with respect to each channel?</p>

<p>I saw that in the description of Tensorflow's <a href="https://www.tensorflow.org/api_docs/python/tf/layers/batch_normalization" rel="noreferrer">tf.layers.batch_normalization</a> it is suggested to perform bn with respect to the channels, but if I recall correctly, I have used the other approach with good results.</p>