<p>During training, the BatchNorm-layer tries to do two things:</p>

<ul>
<li>estimate the mean and variance of the entire training set (population statistics)</li>
<li>normalize the inputs mean and variance, such that they behave like a Gaussian</li>
</ul>

<p>In the ideal case, one would use the population statistic of the entire dataset in the second point. However, these are unknown and change during training. There are also some other issues with this.</p>

<p>A <strong>work-around</strong> is doing the normalization of the input by</p>

<pre><code>gamma * (x - mean) / sigma + b
</code></pre>

<p>based on <strong>mini-batch</strong> statistics <code>mean</code>, <code>sigma</code>.</p>

<p>During training, the running average of mini-batch statistics is used to approximate the <strong>population</strong> statistics.</p>

<p>Now, the original BatchNorm formulation uses the approximated mean and variance of the <strong>entire</strong> dataset for normalization during inference. As the network is fixed, the approximation of the <code>mean</code> and <code>variance</code> should be pretty good. While it seems to make sense to now use the population statistic, it is a critical change: from mini-batch statistics to statistics of the entire training data.</p>

<p>It is critical when batches are not iid or have very small batch sizes during training. (But I also observed it for batches of size 32).</p>

<p>The proposed BatchNorm implicitly simply assumes that both statistics are very similar. In particular, training on mini-batches of size 1 as in pix2pix or dualgan gives very bad information about the population statistic. Here it is the case, that they might contain totally different values.</p>

<p>Having now a deep network, the late layers expect inputs to be normalized batches (in the sense of mini-batch statistics). Note, they are trained on this particular kind of data. But using the entire-dataset statistics violates the assumption during inference.</p>

<p>How to solve this issue? Either also use mini-batch statistics during inference as in the implementations you mentioned. Or use  <a href="https://arxiv.org/pdf/1702.03275.pdf" rel="noreferrer">BatchReNormalization</a> which introduces 2 additional terms to remove the difference between the mini-batch and population statistics
or simply use InstanceNormalization (for regression tasks), which is, in fact, the same as BatchNorm but treats each example in the batch individually and also does no use population statistics.</p>

<p>I also had this issue during research and now use for the regression task the InstanceNorm layer.</p>
What if Batch Normalization is used in training mode when testing?
<p>Batch Normalization has different behavior in training phase and testing phase.</p>

<p>For example, when using <a href="https://www.tensorflow.org/api_docs/python/tf/contrib/layers/batch_norm" rel="noreferrer">tf.contrib.layers.batch_norm</a> in tensorflow, we should set different value for <code>is_training</code> in different phase. </p>

<p>My <strong>qusetion</strong> is: what if I still set <code>is_training=True</code> when testing? That is to say what if I still use the training mode in testing phase? </p>

<p>The reason why I come up with this question is that, the released code of both <a href="https://github.com/affinelayer/pix2pix-tensorflow/blob/master/pix2pix.py#L116" rel="noreferrer">Pix2Pix</a> and <a href="https://github.com/duxingren14/DualGAN/blob/master/ops.py#L8" rel="noreferrer">DualGAN</a> don't set <code>is_training=False</code> when testing. And it seems that if <code>is_training=False</code> is set when testing, the quality of generated images could be very bad.</p>

<p>Is there someone could please explain this? thanks.</p>