title: Tensorflow: save the model with smallest validation error
body: <p>I ran a training job with tensorflow and got the following curve for loss on validation set. The net starts to overfit after 6000-th iteration. So I'd like to get the model before overfitting.</p>
  
  <p><img src="https://www.dropbox.com/s/28esiv862ubjmpr/tensorflow-loss.png?dl=1" alt="loss"></p>
  
  <p>My training code is something like below:</p>
  
  <pre><code>train_step = ......
  summary = tf.scalar_summary(l1_loss.op.name, l1_loss)
  summary_writer = tf.train.SummaryWriter("checkpoint", sess.graph)
  saver = tf.train.Saver()
  for i in xrange(20000):
      batch = get_next_batch(batch_size)
      sess.run(train_step, feed_dict = {x: batch.x, y:batch.y})
      if (i+1) % 100 == 0:
          saver.save(sess, "checkpoint/net", global_step = i+1)
          summary_str = sess.run(summary, feed_dict=validation_feed_dict)
          summary_writer.add_summary(summary_str, i+1)
          summary_writer.flush()
  </code></pre>
  
  <p>After training finishes, there is only five checkpoints saved (19600, 19700, 19800, 19900, 20000). Is there any way to let tensorflow save checkpoint according to the validation error?</p>
  
  <p>P.S. I know that <code>tf.train.Saver</code> has a <code>max_to_keep</code> argument, which in principal could save all the checkpoints. But that's not I wanted (unless it's the only option). I want the saver keep the checkpoint with the smallest validation loss so far. Is that possible?</p>