title: Tensorflow: Setting allow_growth to true does still allocate memory of all my GPUs
body: <p>I have several GPUs but I only want to use one GPU for my training. I am using following options:</p>
  
  <pre><code>config = tf.ConfigProto(allow_soft_placement=True, log_device_placement=True)
  config.gpu_options.allow_growth = True
  
  with tf.Session(config=config) as sess:
  </code></pre>
  
  <p>Despite setting / using all these options, all of my GPUs allocate memory and </p>
  
  <p><code>#processes = #GPUs</code></p>
  
  <p>How can I prevent this from happening?</p>
  
  <p><strong>Note</strong></p>
  
  <ol>
  <li>I do not want use set the devices manually and I do not want to set <code>CUDA_VISIBLE_DEVICES</code> since I want tensorflow to automatically find the best (an idle) GPU available</li>
  <li>When I try to start another <code>run</code> it uses the same GPU that is already used by another tensorflow process even though there are several other free GPUs (apart from the memory allocation on them)</li>
  <li>I am running tensorflow in a docker container: <code>tensorflow/tensorflow:latest-devel-gpu-py</code></li>
  </ol>