title: Model Training accuracy decreases and stops learning after applying Tensorflow's batch normalization
body: <p>So I am working on making a model to classify the <a href="http://benchmark.ini.rub.de/?section=gtsrb&amp;subsection=dataset" rel="nofollow noreferrer">German traffic signs dataset</a> and everything is fun and games. I build my model of 3 <strong>convolutional</strong> layers, 2 <strong>fully-connected(dense)</strong> layers, and 1 last <strong>fully-connected(linear)</strong> as the <em>logits</em> using <strong>tensorflow</strong>, I added <code>dropout</code> with keep prob. of 0.5 to the fully-connected layers and <code>Relu</code> activate for all but the the <code>maxpooling</code> and last <code>fully-connected(logits)</code> layer. </p>
  
  <p><code>conv Relu + maxpool &gt; conv Relu + maxpool &gt; conv  Relu + maxpool &gt; dense + Relu &gt; dense + Relu &gt; dense(logits)</code></p>
  
  <p>This model with deep filters for the convolutional layers works great with validation accuracy as high as 97% and training accuracy thats close to 100%, so as a good practice I decided to add <strong>batch normalization</strong> for the benefits it yields; However, when I add <code>tf.layers.batch_normalization()</code> <em>anywhere with any order</em> my training accuracy dips immediately below the validation accuracy and the model seizes to learn.</p>
  
  <p><a href="https://i.stack.imgur.com/jDfEI.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/jDfEI.png" alt="enter image description here"></a></p>