title: Global step for GAN
body: <p>I'm training a GAN using distributed TensorFlow. </p>
  
  <p>In a typical GAN, we will optimize G and D alternately, while in a distributed training, global step will pass into both G optimizer and D optimizer, then global step will be counted twice in one iteration but acctually we want to count only once.</p>
  
  <p>Here is a part of my code:</p>
  
  <pre><code># build model..
  model = GAN(FLAGS)
  # get G and D optimizer
  g_opt = tf.train.AdamOptimizer(FLAGS.lr, FLAGS.beta1, FLAGS.beta2,
                                     name="g_Adam")
  d_opt = tf.train.AdamOptimizer(FLAGS.lr, FLAGS.beta1, FLAGS.beta2,
                                     name="d_Adam")
  global_step = tf.train.get_or_create_global_step()
  # my question will focus on passing global step into optimizer.minimize()
  g_train_opt = g_opt.minimize(model.g_loss, global_step, model.g_vars)
  d_train_opt = d_opt.minimize(model.d_loss, global_step, model.d_vars)
  # get hooks...
  hooks = [stop_hook, ckpt_hook, sum_hook]
  
  if is_chief:
      session_creator = tf.train.ChiefSessionCreator(
          master=server.target, config=sess_config)
  else:
      session_creator = tf.train.WorkerSessionCreator(
          master=server.target, config=sess_config)
      hooks = None
  
  with tf.train.MonitoredSession(session_creator, hooks=hooks) as mon_sess:
  
      # training
      local_step = 0
      try: 
          while not mon_sess.should_stop():
              t0 = time.time()
              gstep = mon_sess.run(global_step)
              # G and D will be trained alternately
              mon_sess.run(g_train_opt)
              mon_sess.run(d_train_opt)
              delta = time.time() - t0
              g_loss, d_loss = mon_sess.run([model.g_loss, model_d_loss])
              local_step += 1
              print("delta: {}".format(delta))
              print("global_step: {}, local_step: {}".format(gstep, local_step))
              print("g_loss: {} d_loss: {}".format(g_loss, d_loss))
              print("#" * 50)
      except KeyboardInterrupt:
          print("Interrupted")
  </code></pre>
  
  <p>As a result, the global step will be counted twice on both G and D.</p>