title: GPU does not work on the distributed tensorflow cluster
body: <p>I am testing distributed tensorflow using the official code, I have two servers with one K80 GPU and one M40 GPU on them, but running the code on the clusters, only the worker that is specified "is_chief=True" can use the GPU to compute, and the other one is just using CPU, even I have specified the GPU device on it. Part of my code is like below:</p>
  
  <pre><code>def main(_):
  worker_hosts = ['ipaddress0:2222', 'ipaddress1:2222']
  cluster = tf.train.ClusterSpec({"worker": worker_hosts})
  server = tf.train.Server(cluster, job_name='worker', task_index=FLAGS.task_index)
  if FLAGS.job_name == "ps":
      server.join()
  elif FLAGS.job_name == "worker":
      mnist=input_data.read_data_sets('data', one_hot=True)
      # Assigns ops to the local worker by default.
      with tf.device(tf.train.replica_device_setter(worker_device="/job:worker/task:%d/gpu:0" % (FLAGS.task_index), cluster=cluster)):
          print('building model...')
          global_step = tf.contrib.framework.get_or_create_global_step()
          train_step, cross_entropy, accuracy, y, x, y_, keep_prob=build_model(global_step)
          init=tf.global_variables_initializer()
  
      hooks=[tf.train.StopAtStepHook(last_step=1000000)]
      with tf.train.MonitoredTrainingSession(master=server.target, 
          is_chief=(FLAGS.task_index == 0), 
          checkpoint_dir="/tmp/train_logs", 
          hooks=hooks,
          ) as mon_sess:
          mon_sess.run(init)
          i=0
          while not mon_sess.should_stop() and i&lt;=20000:
              i+=1
              batch = mnist.train.next_batch(50)
              if i%50 == 0:
                  train_accuracy= mon_sess.run(accuracy, feed_dict={x:batch[0], y_: batch[1], keep_prob: 1.0})
                  print ("step %d, training accuracy %g"%(mon_sess.run(global_step), train_accuracy))
              mon_sess.run(train_step, feed_dict={x: batch[0], y_: batch[1], keep_prob: 0.5})
          print ("test accuracy %g"%mon_sess.run(accuracy, feed_dict={x:batch[0], y_: batch[1], keep_prob: 1.0}))
  </code></pre>
  
  <p>And the server of ipaddress0 with task_index 0 can use the K80 GPU to compute, the other is using CPU. Does anyone know how to resolve this problem? Please offer some help, THANKS</p>