<p>I read the recommended papers in the answer and comments from
<a href="https://stackoverflow.com/a/40295999/8625228">https://stackoverflow.com/a/40295999/8625228</a></p>
<p>From Ioffe and Szegedy (2015)’s point of view, only use BN in the
network structure. Li et al. (2018) give the statistical and
experimental analyses, that there is a variance shift when the
practitioners use Dropout before BN. Thus, Li et al. (2018) recommend
applying Dropout after all BN layers.</p>
<p>From Ioffe and Szegedy (2015)’s point of view, BN is located
<strong>inside/before</strong> the activation function. However, Chen et al. (2019)
use an IC layer which combines dropout and BN, and Chen et al. (2019)
recommends use BN after ReLU.</p>
<p>On the safety background, I use Dropout or BN only in the network.</p>


<p>Chen, Guangyong, Pengfei Chen, Yujun Shi, Chang-Yu Hsieh, Benben Liao,
and Shengyu Zhang. 2019. “Rethinking the Usage of Batch Normalization
and Dropout in the Training of Deep Neural Networks.” <em>CoRR</em>
abs/1905.05928. <a href="http://arxiv.org/abs/1905.05928" rel="noreferrer">http://arxiv.org/abs/1905.05928</a>.</p>


<p>Ioffe, Sergey, and Christian Szegedy. 2015. “Batch Normalization:
Accelerating Deep Network Training by Reducing Internal Covariate
Shift.” <em>CoRR</em> abs/1502.03167. <a href="http://arxiv.org/abs/1502.03167" rel="noreferrer">http://arxiv.org/abs/1502.03167</a>.</p>


<p>Li, Xiang, Shuo Chen, Xiaolin Hu, and Jian Yang. 2018. “Understanding
the Disharmony Between Dropout and Batch Normalization by Variance
Shift.” <em>CoRR</em> abs/1801.05134. <a href="http://arxiv.org/abs/1801.05134" rel="noreferrer">http://arxiv.org/abs/1801.05134</a>.</p>
Ordering of batch normalization and dropout?
<p><em>The original question was in regard to TensorFlow implementations specifically. However, the answers are for implementations in general. This general answer is also the correct answer for TensorFlow.</em></p>

<p>When using batch normalization and dropout in TensorFlow (specifically using the contrib.layers) do I need to be worried about the ordering?</p>

<p>It seems possible that if I use dropout followed immediately by batch normalization there might be trouble. For example, if the shift in the batch normalization trains to the larger scale numbers of the training outputs, but then that same shift is applied to the smaller (due to the compensation for having more outputs) scale numbers without dropout during testing, then that shift may be off. Does the TensorFlow batch normalization layer automatically compensate for this? Or does this not happen for some reason I'm missing?</p>

<p>Also, are there other pitfalls to look out for in when using these two together? For example, assuming I'm using them in the correct order in regards to the above (assuming there <em>is</em> a correct order), could there be trouble with using both batch normalization and dropout on multiple successive layers? I don't immediately see a problem with that, but I might be missing something.</p>

<p>Thank you much!</p>

<p><strong>UPDATE:</strong></p>

<p>An experimental test <em>seems</em> to suggest that ordering <em>does</em> matter. I ran the same network twice with only the batch norm and dropout reverse. When the dropout is before the batch norm, validation loss seems to be going up as training loss is going down. They're both going down in the other case. But in my case the movements are slow, so things may change after more training and it's just a single test. A more definitive and informed answer would still be appreciated.</p>