title: Keras training with shuffled tf.data: if training is interrupted, how to continue training at last data iteration/order of last saved checkpoint
body: <p>I am training with keras <code>model.fit</code>, and the data comes from tf.records, loaded into a tf.data object, which uses <code>.shuffle</code> to shuffle the data. I am also using <code>callbacks.ModelCheckpoint</code> to save the model every <code>x</code> number of steps/batches.</p>
  <p>Sometimes my cloud instance disconnects or crashes before an epoch is finished, but the model at <code>y</code> step is saved into my drive.</p>
  <p>I would like to finish training over the data in that epoch (I have very long epochs), before training another epoch, so each that each data example is trained over once per epoch.</p>
  <p>Is there a way to get the original order of the data, and the place within the data where model was last saved?</p>
  <h3>What I have found so far</h3>
  <p>It looks like you can set a specific order in .shuffle by setting the seed. However, shuffling only occurs in the buffer, so I am not 100% sure if setting the seed will perfectly reproduce the order. Also, I am not sure how that will work with <code>reshuffle_each_iteration</code>. Is a different seed used after each epoch? If so, I guess a work around is train only 1 epoch at a time, with a specified seed for each epoch.</p>
  <p>Even if I do get a replica of the training order, I'm not sure how to find where in the order was the model last saved, and then to start training from that point. One idea I have to get to the order, is iterate through the dataset manually until I reach it. Although I'm not sure if <code>model.fit()</code> would continue from this order, or start all over. F</p>
  <p>For getting the step/batch number from where the model was last saved, I could probably log this somewhere.</p>
  <p>These solutions seem like rough workarounds, and I am wondering if there's some features in Keras that I may be overlooking to help with this.</p>