<p>The original <a href="https://arxiv.org/abs/1502.03167" rel="noreferrer">batch-norm paper</a> prescribes using the batch-norm before ReLU activation. But there is evidence that it's probably better to use batchnorm <em>after</em> the activation. Here's a comment on <a href="https://github.com/fchollet/keras/issues/1802#issuecomment-187966878" rel="noreferrer">Keras GitHub</a> by Francois Chollet:</p>

<blockquote>
  <p>... I can guarantee that recent code written by Christian [Szegedy] 
  applies relu
  before BN. It is still occasionally a topic of debate, though.</p>
</blockquote>

<p>To your second question: in tensorflow, you can use a high-level <a href="https://www.tensorflow.org/api_docs/python/tf/layers/batch_normalization" rel="noreferrer"><code>tf.layers.batch_normalization</code></a> function, or a low-level <a href="https://www.tensorflow.org/api_docs/python/tf/nn/batch_normalization" rel="noreferrer"><code>tf.nn.batch_normalization</code></a>.</p>
Where to apply batch normalization on standard CNNs
<p>I have the following architecture:</p>

<pre><code>Conv1
Relu1
Pooling1
Conv2
Relu2
Pooling3
FullyConnect1
FullyConnect2
</code></pre>

<p>My question is, where do I apply batch normalization? And what would be the best function to do this in TensorFlow?</p>