title: performance of distributed tensorflow is bad and found high cpu-sys (long time wait on futex)
body: <p>all,I am trying to run an NN model with distributed-tensorflow(2ps+2 worker,each on a standalone machine),the performance is bad,and worker's cpu-usage is between 50%~800% (machine has 40 cores)</p>
  
  <ol>
  <li>I am pretty sure the input queue is ok(without starving problem).</li>
  <li>I also have tried the "tcmlloc/lib/libtcmalloc.so",but no help</li>
  <li>with profile tools "strace",I found something strange:long time wait on futex,sometimes even 2+ seconds wait on futex
  <a href="https://i.stack.imgur.com/bnFqj.jpg" rel="nofollow noreferrer">enter image description here</a></li>
  <li>I also get the timeline info<a href="https://i.stack.imgur.com/2NNtQ.png" rel="nofollow noreferrer">enter image description here</a></li>
  <li>however,When I run the same programe with no-distributed mode,I can achieved almost 4-5 times of performance compared with the distributed mode,and the cpu-usage almost 2000%~3000%</li>
  <li><p>my code is:`    with tf.device(tf.train.replica_device_setter(worker_device = "/job:worker/task:%d" % FLAGS.task_id,cluster=cluster)):</p>
  
  <pre><code>global_step = tf.contrib.framework.get_or_create_global_step()
  
  indexes, values, labels = inputs.inputs()
  logits = models.inference([indexes, values])
  with tf.name_scope('loss'):
      diff = labels * tf.log(logits)
      with tf.name_scope('total'):
          loss = -tf.reduce_mean(diff)
  
  tf.summary.scalar('loss', loss)
  with tf.name_scope('train'):
      print("learning_rate = %f" % FLAGS.learning_rate)
      #sync_opt = tf.train.SyncReplicasOptimizer(tf.train.AdamOptimizer(FLAGS.learning_rate),
                                                #replicas_to_aggregate=1, total_num_replicas=1, use_locking=False)
      sync_opt = tf.train.AdamOptimizer(FLAGS.learning_rate)
      train_step = sync_opt.minimize(loss, global_step=global_step)
  
  with tf.name_scope('accuracy'):
      with tf.name_scope('correct_prediction'):
          correct_prediction = tf.equal(tf.argmax(logits, 1), tf.argmax(labels, 1))
      with tf.name_scope('accuracy'):
          accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))
  tf.summary.scalar('accuracy', accuracy)
  hooks = [tf.train.StopAtStepHook(num_steps=6000)]
  sess_config = tf.ConfigProto(allow_soft_placement=True, log_device_placement=True)
  
  with tf.train.MonitoredTrainingSession(master=server.target,
                                          is_chief=if_chief,
                                          checkpoint_dir=None,
                                          hooks=hooks,config=sess_config,stop_grace_period_secs=20) as session:
  
  
      step = 0
      while not session.should_stop():
          start_time = time.time()
  
          run_metadata = tf.RunMetadata()
          _, global_step_value,loss_value = session.run([train_step, global_step,loss],options=tf.RunOptions(trace_level=tf.RunOptions.FULL_TRACE),
          run_metadata=run_metadata)
          if step &gt; 0 and step % 1 == 0:
  
              duration = time.time() - start_time
              format_str = ("After %d traning steps (%d global steps), "
                            "loss on training batch is %3f. "
                            "(cost %d s)")
              print(format_str % (step, global_step_value, loss_value,duration))
          step += 1
  
          if step &gt;0 and step % 10 == 0 :
               tl = timeline.Timeline(run_metadata.step_stats)
               ctf = tl.generate_chrome_trace_format()
               with open('timeline.json', 'w') as f:
                   f.write(ctf)`
  </code></pre></li>
  </ol>
  
  <p>Any suggestions? Thanks!</p>