title: Nvidia GPU error while using tensorflow
body: <p>Tensorflow session creation fails on GPU node with the below error:</p>
  
  <blockquote>
    <p>2018-06-19 07:01:08.400165: E
    tensorflow/core/common_runtime/direct_session.cc:154] Internal: failed
    initializing StreamExecutor for CUDA device ordinal 0: Internal:
    failed call to cuDevicePrimaryCtxRetain: CUDA_ERROR_ECC_UNCORRECTABLE</p>
  </blockquote>
  
  <p>Below is the GPU info</p>
  
  <pre><code>+-----------------------------------------------------------------------------+
  | NVIDIA-SMI 390.30                 Driver Version: 390.30                    |
  |-------------------------------+----------------------+----------------------+
  | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
  | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
  |===============================+======================+======================|
  |   0  Tesla K80           Off  | 0000752C:00:00.0 Off |                    2 |
  | N/A   39C    P8    25W / 149W |      0MiB / 11441MiB |      0%      Default |
  +-------------------------------+----------------------+----------------------+
  </code></pre>
  
  <p>Please share some pointers to debug this further.</p>
  
  <p>PS: Same program runs fine on CPU node</p>