<blockquote>
<p>when batch_size = 1, variance will be 0</p>
</blockquote>
<p>No, because when you compute mean and variance for BN (for example using <code>tf.nn.monents</code>) you will be computing it over axis <code>[0, 1, 2]</code> (assuming you have NHWC tensor channels order).</p>
<p>From &quot;Group Normalization&quot; paper:
<a href="https://arxiv.org/pdf/1803.08494.pdf" rel="nofollow noreferrer">https://arxiv.org/pdf/1803.08494.pdf</a>
<a href="https://i.stack.imgur.com/8fokp.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/8fokp.png" alt="enter image description here" /></a></p>
<p>With batch_size=1 batch normalization is equal to instance normalization and it can be helpful in some tasks.</p>
<p>But if you are using sort of encoder-decoder and in some layer you have tensor with spatial size of 1x1 it will be a problem, because each channel only have only one value and mean of value will be equal to this value, so BN will zero out information.</p>
Batch normalization when batch size=1
<p>What will happen when I use batch normalization but set <code>batch_size = 1</code>?</p>

<p>Because I am using 3D medical images as training dataset, the batch size can only be set to 1 because of GPU limitation. Normally, I know, when <code>batch_size = 1</code>, variance will be 0. And <code>(x-mean)/variance</code> will lead to error because of division by 0.</p>

<p>But why did errors not occur when I set <code>batch_size = 1</code>? Why my network was trained as good as I expected? Could anyone explain it?</p>

<p><a href="https://stackoverflow.com/questions/59648509/batch-normalization-when-batch-size-1?noredirect=1#comment105458690_59648509">Some people</a> argued that:</p>

<blockquote>
  <p>The <code>ZeroDivisionError</code> may not be encountered because of two cases. First, the exception is caught in a <code>try</code> catch block. Second, a small rational number is added ( <code>1e-19</code> ) to the variance term so that it is never zero. </p>
</blockquote>

<p>But <a href="https://stackoverflow.com/questions/59651396/how-to-calculate-batch-normalization-with-python/59654494?noredirect=1#comment105470721_59654494">some people</a> disagree. They said that:</p>

<blockquote>
  <p>You should calculate mean and std across all pixels in the images of the batch. (So even <code>batch_size = 1</code>, there are still a lot of pixels in the batch. So the reason why <code>batch_size=1</code> can still work is not because of <code>1e-19</code>)</p>
</blockquote>

<p>I have checked the Pytorch source code, and from the code I think the latter one is right. </p>

<p>Does anyone have different opinion???</p>