<p>As a complement to <a href="https://stackoverflow.com/a/41270097/6281477">Prune's answer</a>, during testing, batch normalization layer will use the average <code>mean/variance/scale/shift</code> values from different training iterations to normalize its input(subtract mean and divide by the standard deviation). </p>

<p>And the original <a href="https://arxiv.org/abs/1502.03167" rel="nofollow noreferrer">google's batch normalization paper</a> only said that it should be a moving average method and no more thorough explanation was provided though. Both caffe and tensorflow use an exponential moving average method.</p>

<p>In my experience, a <a href="https://en.wikipedia.org/wiki/Moving_average#Simple_moving_average" rel="nofollow noreferrer">simple moving average method</a> usually better than an exponential moving average method, as far as to the validation accuracy(Maybe it need more experiments). For a compare, you can refer to <a href="https://github.com/knsong/caffe-mt" rel="nofollow noreferrer">here</a>(I tried the two moving average methods implementations in <code>channel_wise_bn_layer</code>, compared with the <code>batch norm</code> layer in <a href="https://github.com/BVLC/caffe" rel="nofollow noreferrer">BVLC/caffe</a>). </p>
What is "batch normalizaiton"? why using it? how does it affect prediction?
<p>Recently, many deep architectures use "batch normalization" for training.</p>

<p>What is "batch normalization"? What does it do mathematically? In what way does it help the training process?</p>

<p>How is batch normalization used during training? is it a special layer inserted into the model? Do I need to normalize before each layer, or only once? </p>

<p>Suppose I used batched normalization for training. Does this affect my test-time model? Should I replace the batch normalization with some other/equivalent layer/operation in my "deploy" network?</p>

<hr>

<p><a href="https://stackoverflow.com/q/29979251/1714410">This question</a> about batch normalization only covers part of this question, I was aiming and hoping for a more detailed answer. More specifically, I would like to know how training with batch normalization affect test time prediction, i.e., the "deploy" network and the TEST phase of the net.</p>