title: MPI CUDA error code 38 cudaErrorNoDevice on Windows
body: <p>I have two mpi processes <strong>server</strong> and <strong>client</strong> which communicate via MPI. A server can handle multiple clients.  I am using MPICH 1.4 and CUDA 6.5 on Windows 7 x64 machine. </p>
  
  <p>On my localhost, I usually run one client instance and one server instance.</p>
  
  <p><strong>When I execute the following command through a command prompt started with administrator privileges:</strong></p>
  
  <blockquote>
    <p>mpiexec.exe -n 1 Server.exe : -n 1 Client.exe</p>
  </blockquote>
  
  <p><strong>the processes do start executing, but after a while I get the following CUDA  error:</strong></p>
  
  <blockquote>
    <p>CUDA error at E:\github\project\src\cudahelper.cpp:71
    code=38(cudaErrorNoDevice) "cudaGetDeviceCount( &amp;_devCount )"</p>
  </blockquote>
  
  <p>I then re-configured smpd and mpiexec using following <a href="http://swash.sourceforge.net/online_doc/swashimp/node9.html" rel="nofollow">guide</a> with my latest domain user account and password credentials. I then double checked in Services and restarted smpd service, but I continue to get the same error.</p>
  
  <p>After some R&amp;D I figured out that if I instead run the mpi processes as <strong>localonly</strong> using the following command, then I get NO CUDA errors:</p>
  
  <blockquote>
    <p>mpiexec.exe <strong>-localonly</strong> -n 1 Server.exe : <strong>-localonly</strong> -n 1 Client.exe</p>
  </blockquote>
  
  <p>Any ideas what is causing CUDA errors when launching MPI processes as non-local processes.</p>