title: Loss value jumps when trained model is saved and restored with tf.train.Saver()
body: <p><strong>EDIT:</strong></p>
  
  <p>The original title of this post was</p>
  
  <p>"Loss value jumps when learning rate is decreased, then comes back down slowly"</p>
  
  <p>however I now think this happens when the saved model is restored, regardless of whether or not the learning rate is changed or not.</p>
  
  <p>After 10 epochs with a learning rate of 0.001, the loss reached ~10. I then saved and restored the model, recommenced training and the loss for the 11th epoch was ~15. It did not reach 10 again for another 5 epochs.</p>
  
  <hr>
  
  <p>I have built an Autoencoder model as shown below. It uses <code>tf.train.Saver()</code> to save and restore the trained model to/from memory. I have the learning rate as a <code>tf.placeholder</code>, thus allowing it to be specified in the <code>feed_dict</code>.</p>
  
  <p>When training the model, I initially begin with a higher learning rate, let it train until the loss begins to plateau, then save the model, manually reduce the learning rate, restore it and recommence training.</p>
  
  <p>However, when this change happens, the loss always jumps far higher than it as prior to the change, and then slowly but consistently reduces again. I cannot understand why this jump would happen, unless there is something wrong with my implementation.</p>
  
  <pre><code>import os
  import pickle as pk
  
  import tensorflow as tf
  
  
  class Autoencoder:
  
      def __init__(self, encoderDims, sparseInput=False, tiedWeights=False, denoise=False):
          self.encoderDims = encoderDims
          self.decoderDims = list(reversed(encoderDims))
          self.sparseInput = sparseInput
          self.tiedWeights = tiedWeights
          self.denoise = denoise          # Only works for greyscale image data
  
          self.input = tf.placeholder(tf.float32, [None, encoderDims[0]])
          self.learningRate = tf.placeholder(tf.float32, [])
  
          self.activationFunction = tf.nn.sigmoid             # TO DO: Allow to be specified by user
          # self.activationFunction = tf.tanh
          # self.activationFunction = tf.nn.selu
          self.SGD = tf.train.AdamOptimizer(self.learningRate)
  
          if self.denoise:
              self.__addNoise()
          self.__buildNetwork()           # Constructs Encoder &amp; Decoder
          self.__buildTensorFlowGraph()   # Creates sequential TensorFlow operations
  
          self.session = tf.Session()
          self.session.run(tf.global_variables_initializer())  # Initialise weights &amp; biases
          self.saver = tf.train.Saver()
          self.session.graph.finalize()   # Avoids memory leaks through duplicating graph nodes
  
      def __addNoise(self):
          # Create a tensor of random numbers with unit variance
          # Then sets pixels to black where values of random tensor &gt; 1
          # (i.e. all values outside the std dev -&gt; ~32% of pixels)
          random = tf.random_normal(tf.shape(self.input))
          mask = tf.greater(random, 1.0)
          self.noisyInput = tf.where(
              mask, tf.ones_like(self.input) * 255, self.input)
  
      def __buildNetwork(self):
          # Lists of weights and biases per layer of encoder and decoder
          self.encoderWeights, self.encoderBiases = [], []
          self.decoderWeights, self.decoderBiases = [], []
          for layer in range(len(self.encoderDims) - 1):
              self.encoderWeights.append(
                  tf.Variable(tf.random_normal(
                      [self.encoderDims[layer], self.encoderDims[layer + 1]]))
              )
              self.encoderBiases.append(
                  tf.Variable(tf.zeros([self.encoderDims[layer + 1]]))
              )
              # if layer != len(self.decoderDims) - 2:  # BIAS IN OUTPUT LAYER????
              self.decoderBiases.append(
                  tf.Variable(tf.zeros([self.decoderDims[layer + 1]]))
              )
              if not self.tiedWeights:
                  self.decoderWeights.append(
                      tf.Variable(tf.random_normal(
                          [self.decoderDims[layer], self.decoderDims[layer + 1]]))
                  )
          if self.tiedWeights:
              self.decoderWeights = [tf.transpose(
                  i) for i in reversed(self.encoderWeights)]
  
      def __buildTensorFlowGraph(self):
          self.encoded = self.encode()        # Encoded/compressed data
          self.decoded = self.decode()        # Decoded/reconstructed data
          self.loss = self.__calculateLoss()
          self.train = self.SGD.minimize(self.loss)
  
      def encode(self):
          if self.denoise:
              encoded = self.noisyInput
          else:
              encoded = self.input
          for layer in range(len(self.encoderDims) - 1):
              encoded = tf.matmul(encoded, self.encoderWeights[layer])
              encoded = tf.add(encoded, self.encoderBiases[layer])
              # if layer != len(self.encoderDims) - 2:    # KEEP LAST LINEAR?
              encoded = self.activationFunction(encoded)
          return encoded
  
      def decode(self):
          decoded = self.encoded
          for layer in range(len(self.decoderDims) - 1):
              decoded = tf.matmul(decoded, self.decoderWeights[layer])
              # if layer != len(self.decoderDims) - 2:  # BIAS IN OUTPUT LAYER????
              decoded = tf.add(decoded, self.decoderBiases[layer])
              if layer != len(self.decoderDims) - 2:  # Keep output layer linear
                  decoded = self.activationFunction(decoded)
          return decoded
  
      def __calculateLoss(self):
          # TO DO: ADD REGULARISATION
          if self.sparseInput:
              nonZeros = tf.where(tf.greater(self.input, 0))
              input = tf.gather(self.input, nonZeros)
              output = tf.gather(self.decoded, nonZeros)
          else:
              input = self.input
              output = self.decoded
  
          return tf.sqrt(
              tf.losses.mean_squared_error(
                  labels=input,
                  predictions=output
              )
          )
  
      def setBatch(self, input, learningRate=0.0):
          self.batchDict = {
              self.input: input,
              self.learningRate: learningRate
          }
  
      def run(self, operations=None, train=False):
          # Returns values of specified list of operations
          # Trains network's parameters if specified
          if not type(operations) is list:
              operations = [operations]
  
          if train:
              ops = [self.train]
          else:
              ops = []
  
          if operations is not None:
              for op in operations:
                  if op == 'input':
                      ops.append(self.input)
                  if op == 'noisyInput':
                      ops.append(self.noisyInput)
                  if op == 'encoded':
                      ops.append(self.encoded)
                  if op == 'decoded':
                      ops.append(self.decoded)
                  if op == 'loss':
                      ops.append(self.loss)
  
          if (train and len(ops) == 2) or (not train and len(ops) == 1):
              return self.session.run(ops, self.batchDict)[-1]
          elif train:
              return self.session.run(ops, self.batchDict)[1:]
          else:
              return self.session.run(ops, self.batchDict)
  
      def save(self, epoch, modelName="Autoencoder"):
          modelName += '.ckpt'
          dir = os.path.dirname(os.path.realpath(__file__)) + '/SavedModels/'
          self.saver.save(self.session, dir + modelName)
          loss = self.session.run(self.loss, self.batchDict)
          with open(dir + modelName + '_epoch.pk', 'wb') as epochFile:
              pk.dump(epoch, epochFile)
          with open(dir + modelName + '_loss.pk', 'wb') as lossFile:
              pk.dump(loss, lossFile)
  
      def restore(self, modelName="Autoencoder"):
          modelName += '.ckpt'
          dir = os.path.dirname(os.path.realpath(__file__)) + '/SavedModels/'
          self.saver.restore(self.session, dir + modelName)
          with open(dir + modelName + '_epoch.pk', 'rb') as epochFile:
              epoch = pk.load(epochFile)
          with open(dir + modelName + '_loss.pk', 'rb') as lossFile:
              loss = pk.load(lossFile)
          return epoch, loss
  
      def kill(self):
          self.session.close()
  </code></pre>
  
  <p>Here is a how the model is trained:</p>
  
  <pre><code>import os
  
  import numpy as np
  
  import pandas as pd
  from Autoencoder import Autoencoder
  
  loadModel = True
  # loadModel = False
  learningRate = 0.001
  numEpochs = 10000
  batchSize = 1  # Divide into 670
  printStep = 1
  
  projectDir = os.path.dirname(os.path.realpath(__file__))
  
  # (671, 9066)
  original = pd.read_csv(projectDir + '/Data/ratings_small_pivoted.csv').drop('userId', axis=1)
  numSamples = original.shape[0]
  numFeatures = original.shape[1]
  numBatches = numSamples // batchSize
  
  encoderDims = [
      numFeatures,
      numFeatures // 2
  ]
  
  ae = Autoencoder(encoderDims, sparseInput=True)
  
  if loadModel:
      bestEpoch, bestLoss = ae.restore()
  else:
      bestEpoch, bestLoss = 0, 9999
  
  for epoch in range(1, numEpochs - bestEpoch + 1):
      epochLoss = 0
      for batch in range(numBatches):
          batchInput = original[batch * batchSize: (batch + 1) * batchSize]
          ae.setBatch(batchInput, learningRate)
          batchLoss = ae.run(['loss'], train=True)
          epochLoss += batchLoss
      epochLoss /= numBatches
      if epochLoss &lt; bestLoss:
          bestLoss = epochLoss
          ae.save(epoch + bestEpoch)
      if epoch == 1 or epoch % printStep == 0:
          print("EPOCH: {} / {}".format(epoch + bestEpoch, numEpochs))
          print("LOSS:  {} ({})\n".format("%.4f" % epochLoss, "%.4f" % bestLoss))
  </code></pre>
  
  <p><a href="https://drive.google.com/file/d/12e1kUjhTVR7NzzyvLXWY5Gkqs6MBS9WF/view?usp=sharing" rel="nofollow noreferrer">Here is the data I am using.</a></p>
  
  <p>With a learning rate of 0.001, the loss reaches ~1.0 after 100 epochs. I then reduce it to 0.0001 and the loss immediately jumps to ~3.9 on the first epoch, then consistently decreases after this epoch, but very slowly.</p>