title: Why loaded Pytorch model's loss highly increased?
body: <p>I'm trying to train Arcface with reference <a href="https://github.com/wujiyang/Face_Pytorch" rel="nofollow noreferrer">to</a>.</p>
  <p>As far as I know, Arcface requires more than 200 training epochs on CASIA-webface with a large batch size.</p>
  <p>Within 100 epochs of training, I stopped the training for a while because I was needed to use GPU for other tasks. And the checkpoints of the model(Resnet) and margin are saved. Before it was stopped, its loss recorded a value between 0.3~1.0, and training accuracy was mount to 80~95%.</p>
  <p>When I resume the Arcface training by loading the checkpoint files using load_sate, it seems normal when the first batch is processed. But suddenly the loss increased sharply and the accuracy became very low.</p>
  <p>Its loss suddenly became increased. How did this happen? I had no other way so anyway continued the training, but I don't think the loss is decreasing well even though it is a trained model over 100 epochs...</p>
  <p>When I searched for similar issues, they told the problem was that the optimizer was not saved (Because the reference github page didn't save the optimizer, so did I. Is it true?</p>
  <p><a href="https://i.stack.imgur.com/0KiaZ.png" rel="nofollow noreferrer">My losses after loading</a></p>