title: Tensorflow memory leak while using GPU but in CPU is OK
body: <p>While using GPU to train my model, the memory of CPU would exhaust after a few epochs. However, if I use CPU instead, this behaviour wouldn't occur. </p>
  
  <p>I tried a couple of different combinations, but none of them worked:</p>
  
  <ul>
  <li>cuda8.0.44 cudnn5.1 tensorflow1.2.1</li>
  <li>cuda8.0.44 cudnn6.0 tensorflow1.2.1</li>
  <li>cuda8.0.44 cudnn6.0 tensorflow1.3.0</li>
  <li>cuda8.0.61(with patch) cudnn6.0 tensorflow1.3.0</li>
  </ul>
  
  <p>I'm using ubuntu14.0.4 with python2.7</p>
  
  <p>Based on the suggestion in this <a href="https://stackoverflow.com/questions/42861956/gpu-poolallocator-explodes-the-cpu-memory">related question</a>, I already tried the following:</p>
  
  <ol>
  <li><p>Calling <code>tf.get_default_graph().finalize()</code></p></li>
  <li><p>Use <code>tcmalloc</code> instead of <code>malloc</code></p></li>
  <li><p>Recompiled tensorflow with different hardcoded params</p></li>
  </ol>