title: tensorflow on GPU: no known devices, despite cuda's deviceQuery returning a "PASS" result
body: <blockquote>
    <p>Note : this question was initially <a href="https://github.com/tensorflow/tensorflow/issues/7648#issuecomment-280866214" rel="noreferrer">asked on github</a>, but it was asked to be here instead</p>
  </blockquote>
  
  <p>I'm having trouble running tensorflow on gpu, and it does not seems to be the usual cuda's configuration problem, because everything seems to indicate cuda is properly setup.</p>
  
  <p>The main symptom: when running tensorflow, my gpu is not detected (<a href="https://gist.github.com/oelmekki/cafda411bf5c2ea695d984fa98e0995b" rel="noreferrer">the code being run</a>, and <a href="https://gist.github.com/oelmekki/77235c6b0dde99b3438f190eb557f40f" rel="noreferrer">its output</a>).</p>
  
  <p>What differs from usual issues is that cuda seems properly installed and running <code>./deviceQuery</code> from cuda samples is successful (<a href="https://gist.github.com/oelmekki/fe65a15daec45aa90ec33b10b51d3aae" rel="noreferrer">output</a>).</p>
  
  <p>I have two graphical cards:</p>
  
  <ul>
  <li>an old GTX 650 used for my monitors (I don't want to use that one with tensorflow)</li>
  <li>a GTX 1060 that I want to dedicate to tensorflow</li>
  </ul>
  
  <p>I use:</p>
  
  <ul>
  <li><a href="https://pypi.python.org/pypi/tensorflow" rel="noreferrer">tensorflow-1.0.0</a></li>
  <li>cuda-8.0 (<a href="https://gist.github.com/oelmekki/6e5e9d7d1ea871e1d73efae307efe9ce" rel="noreferrer">ls -l /usr/local/cuda/lib64/libcud*</a>)</li>
  <li>cudnn-5.1.10</li>
  <li>python-2.7.12</li>
  <li>nvidia-drivers-375.26 (this was installed by cuda and replaced my distro driver package)</li>
  </ul>
  
  <p>I've tried:</p>
  
  <ul>
  <li>adding <code>/usr/local/cuda/bin/</code> to <code>$PATH</code></li>
  <li>forcing gpu placement in tensorflow script using <code>with tf.device('/gpu:1'):</code> (and <code>with tf.device('/gpu:0'):</code> when it failed, for good measure)</li>
  <li>whitelisting the gpu I wanted to use with <code>CUDA_VISIBLE_DEVICES</code>, in case the presence of my old unsupported card did cause problems</li>
  <li>running the script with sudo (because why not)</li>
  </ul>
  
  <p>Here are the outputs of <a href="https://gist.github.com/oelmekki/7bdcb5cc2f791cea561a60f8b21e87b5" rel="noreferrer">nvidia-smi</a> and <a href="https://gist.github.com/oelmekki/b83a5a0a72e8924aeb44b70b3598f9b4" rel="noreferrer">nvidia-debugdump -l</a>, in case it's useful.</p>
  
  <p>At this point, I feel like I have followed all the breadcrumbs and have no idea what I could try else. I'm not even sure if I'm contemplating a bug or a configuration problem. Any advice about how to debug this would be greatly appreciated. Thanks!</p>
  
  <p><strong>Update</strong>: with the help of Yaroslav on github, I gathered more debugging info by raising log level, but it doesn't seem to say much about the device selection : <a href="https://gist.github.com/oelmekki/760a37ca50bf58d4f03f46d104b798bb" rel="noreferrer">https://gist.github.com/oelmekki/760a37ca50bf58d4f03f46d104b798bb</a></p>
  
  <p><strong>Update 2</strong>: Using theano detects gpu correctly, but interestingly it complains about cuDNN being too recent, then fallback to cpu (<a href="https://gist.github.com/oelmekki/34b6e41a0ff2b17ff9f39bcf56d0635a" rel="noreferrer">code ran</a>, <a href="https://gist.github.com/oelmekki/11626d6b34058337dae64f1915e5a9fe" rel="noreferrer">output</a>). Maybe that could be the problem with tensorflow as well?</p>