<p>After reading the original papers of batch normalization (<a href="https://arxiv.org/abs/1502.03167" rel="nofollow noreferrer">https://arxiv.org/abs/1502.03167</a>) and SELU (<a href="https://arxiv.org/abs/1706.02515" rel="nofollow noreferrer">https://arxiv.org/abs/1706.02515</a>), I have a better understanding of them:</p>

<ol>
<li><p>batch normalization is an "isolation" procedure to ensure the input (in any mini-batch) to the next layer has a fixed distribution, therefore the so called "shifting variance" problem is fixed. The affine transform ( γ*x^ + β ) just tunes the standardized x^ to another <strong>fixed distribution</strong> for better expressiveness. For the simple normalization, we need to turn the <code>center</code> and <code>scale</code> parameters to <code>False</code> when calling <code>tf.layers.batch_normalization</code>.</p></li>
<li><p>Make sure the <code>epsilon</code> (still in <code>tf.layers.batch_normalization</code>) is set to at least 2 magnitudes less than the lowest magnitude of the all input data. The default value of <code>epsilon</code> is set to 0.001. For my case, some features have values as low as 1e-6. Therefore, I had to change <code>epsilon</code> to 1e-8.</p></li>
<li><p>The inputs to SELU have to be normalized before feeding them into the model. <code>tf.layers.batch_normalization</code> is not designed for that purpose.</p></li>
</ol>
Use tf.layers.batch_normalization to preprocess inputs for SELU activation function?
<p>The SELU activation function (<a href="https://github.com/bioinf-jku/SNNs/blob/master/selu.py" rel="nofollow noreferrer">https://github.com/bioinf-jku/SNNs/blob/master/selu.py</a>) requires the input to be normalized to have the mean value of 0.0 and the variance of 1.0. Therefore, I tried to apply <code>tf.layers.batch_normalization</code> (<code>axis=-1</code>) on the raw data to meet that requirement. The raw data in each batch have the shape of <code>[batch_size, 15]</code>, where 15 refers to the number of features. The graph below shows the variances of 5 of these features returned from <code>tf.layers.batch_normalization</code> (~20 epochs). They are not all close to 1.0 as expected. The mean values are not all close to 0.0 as well (graphs not shown). </p>

<p>How should I get the 15 features all normalized independently (I expect every feature after normalization will have mean = 0 and var = 1.0)? </p>

<p><a href="https://i.stack.imgur.com/8EcQg.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/8EcQg.png" alt="enter image description here"></a></p>