<p>First up, there are several ways to apply batch normalization, which are even mentioned in the <a href="https://arxiv.org/pdf/1502.03167v3.pdf" rel="nofollow noreferrer">original paper</a> specifically for convolutional neural networks. See the discussion in <a href="https://stackoverflow.com/q/38553927/712995">this question</a>, which outlines the difference between a <em>usual</em> and <em>convolutional</em> BN, and also the reason why both approaches make sense.</p>

<p>Particularly <a href="https://keras.io/layers/normalization/" rel="nofollow noreferrer"><code>keras.layers.BatchNormalization</code></a> implements the <em>convolutional</em> BN, which means that for an input <code>[m,h,w,c]</code>  it computes <code>c</code> means and standard deviations across <code>m*h*w</code> values. The shapes of the running mean, running std dev and gamma and beta variables are just <code>(c,)</code>. The values across spatial dimensions (<em>pixels</em>), as well as across the batch, are <strong>shared</strong>.</p>

<p>So a more accurate algorithm would be: for each R, G, and B channel compute the mean/variance across all pixels and all images in this channel and apply the normalization.</p>
What is being normalized by Keras/TensorFlow BatchNormalization
<p>My question is what is being normalized by BatchNormalization (BN).</p>

<p>I am asking, does BN normalize the channels for each pixel separately or for all the pixels together. And does it do it on a per image basis or on all the channels of the entire batch.</p>

<p>Specifically, BN is operating on <code>X</code>. Say, <code>X.shape = [m,h,w,c]</code>. So with <code>axis=3</code>, it is operating on the "c" dimension which is the number of channels (for rgb) or the number of feature maps.</p>

<p>So lets say the <code>X</code> is an rgb and thus has 3 channels. Does the BN do the following: (this is a simplified version of the BN to discuss the dimensional aspects.  I understand that gamma and beta are learned but not concerned with that here.)</p>

<p>For each <code>image=X</code> in <code>m</code>:</p>

<ol>
<li>For each pixel (h,w) take the mean of the associated r, g, &amp; b values.</li>
<li>For each pixel (h,w) take the variance of the associated r, g, &amp; b values</li>
<li>Do <code>r = (r-mean)/var</code>, <code>g = (g-mean)/var</code>, &amp; <code>b = (b-mean)/var</code>, where r, g, &amp; b are the red, green, &amp; blue channels of <code>X</code> respectively.</li>
<li>Then repeat this process for the next image in <code>m</code>,</li>
</ol>

<p>In keras, the docs for BatchNormalization says:</p>

<blockquote>
  <p>axis: Integer, the axis that should be normalized (typically the features axis).</p>
  
  <p>For instance, after a <code>Conv2D</code> layer with <code>data_format="channels_first"</code>,
  set <code>axis=1</code> in <code>BatchNormalization</code>.</p>
</blockquote>

<p>But what is it exactly doing along each dimension?</p>