<p><a href="https://arxiv.org/pdf/1502.03167.pdf" rel="nofollow noreferrer">Batch Normalization</a> was introduced to reduce the internal covariate shift of the input feature maps. Due to change of parameters of each layer after every optimization steps, input distribution of a layer also changes, this slow down the model convergence. By using Batch Normalization we can normalize the input distribution irrespective of the batch_size (whether batch_size =1 or larger).</p>
<blockquote>
<p>BN normalizes the input distribution</p>
</blockquote>
<p>For convolutional network input for intermediate layer is 4D tensor. <code>[batch_size, width, height, num_filters]</code>. Normalization effect all the feature maps.</p>
<blockquote>
<p>delete the BN layers (merge the BN layers with the preceding layers for the pretrained model)</p>
</blockquote>
<p>This may further slow down the training step and convergence mayn't be achieved.</p>
<blockquote>
<p>Freeze the parameters and statistics of the BN layers</p>
</blockquote>
<p>Sometime the input data distribution for retrain/finetune, may vary significantly from the original data used to train the pretrained model used for initialization, Due to which your model may end-up in non-optimal solution.</p>
How to handle the BatchNorm layer when training fully convolutional networks by finetuning?
<p>Training fully convolutional nerworks (FCNs) for pixelwise semantic segmentation is very memory intensive. So we often use batchsize=1 for traing FCNs. However, when we finetune the pretrained networks with BatchNorm (BN) layers, batchsize=1 doesn't make sense for the BN layers. So, how to handle the BN layers?</p>

<p>Some options:</p>

<ol>
<li><p>delete the BN layers (merge the BN layers with the preceding layers for the pretrained model)</p></li>
<li><p>Freeze the parameters and statistics of the BN layers</p></li>
<li><p>....</p></li>
</ol>

<p>which is better and any demo for implementation in pytorch/tf/caffe?</p>