title: Tensorflow Object Detection API w/ TPU Training - Display more granular Tensorboard plots
body: <p>I've been following <a href="https://medium.com/tensorflow/training-and-serving-a-realtime-mobile-object-detector-in-30-minutes-with-cloud-tpus-b78971cf1193" rel="nofollow noreferrer">this tutorial</a> on the <a href="https://github.com/tensorflow/models/tree/master/research/object_detection" rel="nofollow noreferrer">Tensorflow Object Detection API</a>, and I've successfully trained my own object detection model using Google's Cloud TPUs.</p>
  
  <p>However, the problem is that on Tensorboard, the plots I'm seeing only have 2 data points each (so it just plots a straight line), like this:
  <a href="https://i.stack.imgur.com/6mSgd.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/6mSgd.png" alt="enter image description here"></a></p>
  
  <p>...whereas I want to see more "granular" plots like these below, which are much more detailed:</p>
  
  <p><a href="https://i.stack.imgur.com/rZdpW.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/rZdpW.png" alt="enter image description here"></a></p>
  
  <p><a href="https://medium.com/tensorflow/training-and-serving-a-realtime-mobile-object-detector-in-30-minutes-with-cloud-tpus-b78971cf1193" rel="nofollow noreferrer">The tutorial I've been following</a> acknowledges that this issue is caused by the fact that TPU training requires very few steps to train:</p>
  
  <blockquote>
    <p>Note that these graphs only have 2 points plotted since the model
    trains quickly in very few steps (if you’ve used TensorBoard before
    you may be used to seeing more of a curve here)</p>
  </blockquote>
  
  <p>I tried adding <code>save_checkpoints_steps=50</code> in the file <code>model_tpu_main.py</code> (see code fragment below), and when I re-ran training, I was able to get a more granular plot, with 1 data point every 300 steps or so.</p>
  
  <pre><code>config = tf.contrib.tpu.RunConfig(
        # I added this line below:
        save_checkpoints_steps=50,
  
        master=tpu_grpc_url,
        evaluation_master=tpu_grpc_url,
        model_dir=FLAGS.model_dir,
        tpu_config=tf.contrib.tpu.TPUConfig(
            iterations_per_loop=FLAGS.iterations_per_loop,
            num_shards=FLAGS.num_shards))
  </code></pre>
  
  <p><a href="https://i.stack.imgur.com/zRBQQ.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/zRBQQ.png" alt="enter image description here"></a></p>
  
  <p>However, my training job is actually saving a checkpoint every 100 steps, rather than every 300 steps. Looking at the logs, my evaluation job is running every 300 steps. Is there a way I can make my evaluation job run every 100 steps (whenever there's a new checkpoint) so that I can get more granular plots on Tensorboard?</p>