<blockquote>
<p>variance will be 0</p>
</blockquote>
<p>No, it won't; <code>BatchNormalization</code> computes statistics only with respect to a <em>single axis</em> (usually the channels axis, <code>=-1</code> (last) by default); every other axis is <em>collapsed</em>, i.e. summed over for averaging; details below.</p>
<p>More importantly, however, unless you can explicitly justify it, I advise against using <code>BatchNormalization</code> with <code>batch_size=1</code>; there are strong theoretical reasons against it, and multiple publications have shown BN performance degrade for <code>batch_size</code> under 32, and severely for &lt;=8. In a nutshell, batch statistics &quot;averaged&quot; over a single sample vary greatly sample-to-sample (high variance), and BN mechanisms don't work as intended.</p>
<p><strong>Small mini-batch alternatives</strong>: <a href="https://arxiv.org/abs/1702.03275" rel="nofollow noreferrer">Batch Renormalization</a> -- <a href="https://arxiv.org/abs/1607.06450" rel="nofollow noreferrer">Layer Normalization</a> -- <a href="https://arxiv.org/abs/1602.07868" rel="nofollow noreferrer">Weight Normalization</a></p>
<hr>
<p><strong>Implementation details</strong>: from <a href="https://github.com/keras-team/keras/blob/master/keras/layers/normalization.py#L137" rel="nofollow noreferrer">source code</a>:</p>
<pre class="lang-py prettyprint-override"><code>reduction_axes = list(range(len(input_shape)))
del reduction_axes[self.axis]
</code></pre>
<p>Eventually, <a href="https://github.com/tensorflow/tensorflow/blob/r2.1/tensorflow/python/ops/nn_impl.py#L1220" rel="nofollow noreferrer"><code>tf.nn.monents</code></a> is called with <code>axes=reduction_axes</code>, which performs a <code>reduce_sum</code> to compute <code>variance</code>. Then, in the TensorFlow backend, <code>mean</code> and <code>variance</code> are <a href="https://github.com/keras-team/keras/blob/master/keras/backend/tensorflow_backend.py#L2191" rel="nofollow noreferrer">passed</a> to <a href="https://github.com/tensorflow/tensorflow/blob/r2.1/tensorflow/python/ops/nn_impl.py#L1373-L1434" rel="nofollow noreferrer"><code>tf.nn.batch_normalization</code></a> to return train- or inference-normalized inputs.</p>
<p>In other words, if your input is <code>(batch_size, height, width, depth, channels)</code>, or <code>(1, height, width, depth, channels)</code>, then BN will run calculations over the <code>1</code>, <code>height</code>, <code>width</code>, and <code>depth</code> dimensions.</p>
<p><strong>Can variance ever be zero?</strong> - yes, if every single datapoint for any given <code>channel</code> slice (along every dimension) is the same. But this should be near-impossible for real data.</p>
<hr>
<p><strong>Other answers</strong>: first one is misleading:</p>
<blockquote>
<p>a small rational number is added (<code>1e-19</code>) to the variance</p>
</blockquote>
<p>This doesn't happen in computing variance, but it is added <em>to</em> variance when normalizing; nonetheless, it is rarely necessary, as <code>variance</code> is far from zero. Also, the epsilon term is actually defaulted to <code>1e-3</code> by Keras; it serves roles in regularizing, beyond mere avoiding zero-division.</p>
<hr>
<p><strong>Update</strong>: I failed to address an important piece of intuition with suspecting variance to be 0; indeed, the <em>batch statistics</em> variance is zero, since there is only <em>one statistic</em> - but the &quot;statistic&quot; itself concerns the mean &amp; variance of the channel + spatial dimensions. In other words, the variance <em>of</em> the mean &amp; variance (<em>of</em> the single train sample) is zero, but the mean &amp; variance themselves aren't.</p>
Batch normalization when batch size=1
<p>What will happen when I use batch normalization but set <code>batch_size = 1</code>?</p>

<p>Because I am using 3D medical images as training dataset, the batch size can only be set to 1 because of GPU limitation. Normally, I know, when <code>batch_size = 1</code>, variance will be 0. And <code>(x-mean)/variance</code> will lead to error because of division by 0.</p>

<p>But why did errors not occur when I set <code>batch_size = 1</code>? Why my network was trained as good as I expected? Could anyone explain it?</p>

<p><a href="https://stackoverflow.com/questions/59648509/batch-normalization-when-batch-size-1?noredirect=1#comment105458690_59648509">Some people</a> argued that:</p>

<blockquote>
  <p>The <code>ZeroDivisionError</code> may not be encountered because of two cases. First, the exception is caught in a <code>try</code> catch block. Second, a small rational number is added ( <code>1e-19</code> ) to the variance term so that it is never zero. </p>
</blockquote>

<p>But <a href="https://stackoverflow.com/questions/59651396/how-to-calculate-batch-normalization-with-python/59654494?noredirect=1#comment105470721_59654494">some people</a> disagree. They said that:</p>

<blockquote>
  <p>You should calculate mean and std across all pixels in the images of the batch. (So even <code>batch_size = 1</code>, there are still a lot of pixels in the batch. So the reason why <code>batch_size=1</code> can still work is not because of <code>1e-19</code>)</p>
</blockquote>

<p>I have checked the Pytorch source code, and from the code I think the latter one is right. </p>

<p>Does anyone have different opinion???</p>