<p>I do not have an exact answer, but here are my thoughts:</p>

<hr>

<blockquote>
  <p>networks with BatchNorm (BN) layers, batchsize=1 doesn't make sense
  for the BN layers</p>
</blockquote>

<p>The main motivation of BN is to fix the distribution (mean/variance) of the input in the batch. In my opinion, having one element this does not make sense. <a href="https://arxiv.org/pdf/1502.03167.pdf" rel="nofollow noreferrer">Judging from the paper</a> <a href="https://i.stack.imgur.com/G0KXa.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/G0KXa.png" alt="enter image description here"></a></p>

<p>you will need to calculate the mean and the variance for 1 element, which does not make sense.</p>

<hr>

<p>You can always just remove BN but are you sure you can't afford at least 16 elements in the batch?</p>
How to handle the BatchNorm layer when training fully convolutional networks by finetuning?
<p>Training fully convolutional nerworks (FCNs) for pixelwise semantic segmentation is very memory intensive. So we often use batchsize=1 for traing FCNs. However, when we finetune the pretrained networks with BatchNorm (BN) layers, batchsize=1 doesn't make sense for the BN layers. So, how to handle the BN layers?</p>

<p>Some options:</p>

<ol>
<li><p>delete the BN layers (merge the BN layers with the preceding layers for the pretrained model)</p></li>
<li><p>Freeze the parameters and statistics of the BN layers</p></li>
<li><p>....</p></li>
</ol>

<p>which is better and any demo for implementation in pytorch/tf/caffe?</p>