title: How to handle the BatchNorm layer when training fully convolutional networks by finetuning?
body: <p>Training fully convolutional nerworks (FCNs) for pixelwise semantic segmentation is very memory intensive. So we often use batchsize=1 for traing FCNs. However, when we finetune the pretrained networks with BatchNorm (BN) layers, batchsize=1 doesn't make sense for the BN layers. So, how to handle the BN layers?</p>
  
  <p>Some options:</p>
  
  <ol>
  <li><p>delete the BN layers (merge the BN layers with the preceding layers for the pretrained model)</p></li>
  <li><p>Freeze the parameters and statistics of the BN layers</p></li>
  <li><p>....</p></li>
  </ol>
  
  <p>which is better and any demo for implementation in pytorch/tf/caffe?</p>