title: TensorFlow: NotFoundError: Key not found in checkpoint
body: <p>I've been training a TensorFlow model for about a week with occasional sessions of fine-tuning.</p>
  
  <p>Today when I tried to finetune the model I got the error:</p>
  
  <pre><code>tensorflow.python.framework.errors_impl.NotFoundError: Key conv_classifier/loss/total_loss/avg not found in checkpoint
   [[Node: save/RestoreV2_37 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_37/tensor_names, save/RestoreV2_37/shape_and_slices)]]
  </code></pre>
  
  <p>Using inspect_checkpoint.py I see that the checkpoint file now has two empty layers in it:</p>
  
  <pre><code>...
  conv_decode4/ort_weights/Momentum (DT_FLOAT) [7,7,64,64]
  loss/cross_entropy/avg (DT_FLOAT) []
  loss/total_loss/avg (DT_FLOAT) []
  up1/up_filter (DT_FLOAT) [2,2,64,64]
  ...
  </code></pre>
  
  <p>How do I fix this problem?</p>
  
  <p>SOLUTION:</p>
  
  <p>Following suggestion by mrry below edited for clarity:</p>
  
  <pre><code>code_to_checkpoint_variable_map = {var.op.name: var for var in tf.global_variables()}
  for code_variable_name, checkpoint_variable_name in {
       "inference/conv_classifier/weight_loss/avg" : "loss/weight_loss/avg",
       "inference/conv_classifier/loss/total_loss/avg" : "loss/total_loss/avg",
       "inference/conv_classifier/loss/cross_entropy/avg": "loss/cross_entropy/avg",
  }.items():
      code_to_checkpoint_variable_map[checkpoint_variable_name] = code_to_checkpoint_variable_map[code_variable_name]
      del code_to_checkpoint_variable_map[code_variable_name]
  
  saver = tf.train.Saver(code_to_checkpoint_variable_map)
  saver.restore(sess, tf.train.latest_checkpoint('./logs'))
  </code></pre>