title: Computer crashes while training with tensorflow on multiple GPUs
body: <p>My servers crash and automatically restart while training with tensorflow using more than one GPUs.</p>
  
  <p>My task is like:</p>
  
  <pre><code>CUDA_VISIBLE_DEVICES=0 python train.py [LSTM training]
  CUDA_VISIBLE_DEVICES=1 python train.py [Another LSTM training]
  </code></pre>
  
  <p>When the tasks are both running, the server crashes and automatically restarts. I tried with different servers, but all servers behave the same.</p>
  
  <p>My hardware configuration:</p>
  
  <p>Dell T7610 Server<br>
  2* E5-2609 CPU<br>
  1* Nvidia GTX 1080 (Inno 3D Foundation Edition)<br>
  1* Nvidia GTX 1080Ti (Gigabyte Foundation Edition)<br>
  64G memory  </p>
  
  <p>My software configuration:</p>
  
  <p>Ubuntu 14.04 LTS (Linux 3.13.0-87-generic)<br>
  Nvidia Driver version 378.13 (nvidia-378 from ppa:graphics-drivers)<br>
  CUDA 8.0 (CuDNN 5.1.5)<br>
  Python 2.7.13 |Anaconda 4.3.0 (64-bit)<br>
  tensorflow-gpu-1.0.1 (from pip install)  </p>
  
  <p>The PSU has enough capacity for the graphics cards. And from memtest results, it's not a memory-related problem. There are not any <code>syslog</code> and <code>kern.log</code> log that helps to locate the problem. </p>
  
  <p>I am aware of <a href="https://stackoverflow.com/questions/41157215/training-tensorflow-on-multiple-gpu-crashes-the-computer?rq=1">a similar question</a>. However, a workaround that downgrades the Linux kernel is unacceptable since I share the machines with lab folks.</p>